MarkLogic XML and JSON Storage, Indexing, and Querying

This article explores how MarkLogic handles enterprise-grade data management for structured and semi-structured XML and JSON documents. It details the underlying storage architecture, the mechanics of MarkLogic’s Universal Indexing, its multi-model query execution engine, and the distributed clustering model that enables high-throughput scalability with full ACID compliance.

Storage Architecture: Forests, Stands, and Immutable Data

MarkLogic treats JSON documents and XML fragments as first-class primitives. Unlike traditional relational databases that require rigid schemas, MarkLogic persists documents as structured hierarchical trees directly onto disk in an optimized binary format.

• Forests and Stands: The database organizes data logically into collections called Forests. Each forest consists of multiple read-only filesystem structures called Stands. When documents are ingested or updated, they are written to an in-memory stand (the in-memory journal) and subsequently flushed to disk as a new immutable stand.
• Append-Only and MVCC: MarkLogic uses Multi-Version Concurrency Control (MVCC). Updates do not overwrite existing records; instead, a new version of the document is written to a new stand, and the old version is marked as deleted via a bit-mapped deletion list. Background merge processes continually consolidate stands, purge deleted documents, and optimize disk space.
• ACID Transactions: Despite using a distributed, document-centric model, MarkLogic ensures full Atomicity, Consistency, Isolation, and Durability (ACID) across all transactions, supporting multi-document and multi-statement transactional integrity.

The Universal Index: Ingestion Without Predefined Schemas

MarkLogic eliminates the need to manually design and maintain indexes for standard operations through its built-in Universal Index.

• Automatic Inversion: During document ingestion, the system parses the structural hierarchy of the XML or JSON document. It automatically indexes words, element/property names, relationships, and values without requiring predefined schema definitions.
• Index Types:
  • Structure and Content Indexes: Map terms to the precise path and hierarchy within documents, supporting both full-text search and structural path navigation.
  • Value and Range Indexes: Highly optimized, in-memory columnar indexes created for specific data types (strings, integers, dates) to support rapid sorting, range filtering, and aggregations.
  • Triple/Graph Indexes: Native indexing of semantic triples (RDF) embedded within documents or stored independently, enabling knowledge-graph capabilities alongside document storage.
  • Geospatial Indexes: Support for spatial queries over coordinate elements embedded within JSON or XML.

Because index entries are created during the write pipeline, read operations bypass the need for full document scans.

Query Processing: Search Meets Database Queries

MarkLogic merges full-text search engine technology with structured query capabilities, executing operations through a two-phase query evaluation process.

1. Index Resolution (Filtered Phase): The query engine evaluates the query criteria strictly against the in-memory indexes. This step returns a candidate list of matching document fragments (Fragment IDs) with extreme speed, eliminating irrelevant data.
2. Document Filtering (Unfiltered Phase): If a query requires deep verification not fully covered by the indexes (e.g., complex wildcards or custom stemmers), the engine inspects the actual persisted document data from the candidate list to guarantee 100% accuracy.

Query Interfaces

Enterprise developers can query MarkLogic using several native paradigms: * Optic API: A unified data access interface enabling relational-style operations (joins, group-bys, aggregates) across documents, triples, and rows. * Server-Side JavaScript and XQuery: Native runtime environments running inside the database layer, minimizing network latency for data-intensive transformations. * SQL and SPARQL: Standard SQL support via ODBC/JDBC drivers for business intelligence tools, and SPARQL for querying semantic RDF graph data. * REST APIs: Standard HTTP interfaces for cross-platform integration and search-style document retrieval.

Scalability and Distributed Clustering

MarkLogic achieves horizontal scalability through a shared-nothing, distributed architecture that scales compute and storage independently.

• E-Nodes and D-Nodes: A MarkLogic cluster separates responsibilities into Evaluator Nodes (E-nodes) and Data Nodes (D-nodes). E-nodes handle client connections, query parsing, and business logic execution, while D-nodes manage disk I/O, forest storage, and local index lookups.
• Rebalancing and Sharding: Forests act as natural shards. As new nodes join the cluster, MarkLogic’s dynamic rebalancer redistributes forests across the infrastructure to balance capacity and query load automatically.
• High Availability and Disaster Recovery: Built-in forest replication replicates data synchronously or asynchronously to replica nodes, ensuring automatic failover without data loss. Local and cross-data-center replication guarantees continuous operations and enterprise disaster recovery.