Elasticsearch Python: Query Distributed Indices with DSL

This article provides an overview of how the official Elasticsearch Python client interacts with a distributed cluster using the JSON-based Query DSL. You will learn how query payloads are structured as native Python dictionaries, how the client communicates these requests to cluster nodes, how Elasticsearch coordinates search operations across distributed shards, and how to execute multi-index queries effectively in Python.

Query DSL and the Python Client

The official Elasticsearch Python client (elasticsearch-py) acts as an HTTP transport wrapper that translates Python objects into RESTful HTTP requests. In Elasticsearch, searches are defined using Query DSL (Domain Specific Language), a JSON-based format for specifying query criteria, aggregations, filters, and scoring logic.

Because Python dictionaries natively mirror JSON structures, queries written in Elasticsearch Query DSL are defined directly as nested Python dictionaries and passed to the search() API method via the query or body parameter.

How Distributed Search Works Under the Hood

When querying an index distributed across multiple nodes and shards, Elasticsearch handles the distributed nature of the data transparently to the client:

  1. Client Request: The Python client sends an HTTP POST request containing the JSON DSL query to a designated node (known as the coordinating node).
  2. Scatter Phase (Query Phase): The coordinating node identifies which shards contain the target indices. It broadcasts the query to one copy of each shard (either a primary or a replica) across the cluster. Each shard executes the query locally, generating a priority queue of matching document IDs and relevance scores.
  3. Gather Phase (Fetch Phase): The coordinating node merges the local priority queues from all shards into a globally sorted result set. It then requests the actual document contents (_source) only for the top-ranking documents matching the size and from parameters.
  4. Response Delivery: The coordinating node packages the documents and execution metadata into a single JSON response, which the Python client parses back into a Python dictionary.

Querying Multiple or Distributed Indices

To query across distributed indices, pass an index pattern, comma-separated list, or wildcard to the index argument of the search method.

from elasticsearch import Elasticsearch

# Initialize the client
client = Elasticsearch(
    "https://localhost:9200",
    basic_auth=("elastic", "your_password"),
    verify_certs=False
)

# Define Query DSL using Python dictionaries
search_query = {
    "bool": {
        "must": [
            {"match": {"status": "active"}}
        ],
        "filter": [
            {"range": {"created_at": {"gte": "now-7d/d"}}}
        ]
    }
}

# Execute query across distributed indices using wildcards
response = client.search(
    index="logs-*,metrics-*",
    query=search_query,
    size=10
)

# Process results
total_hits = response["hits"]["total"]["value"]
hits = response["hits"]["hits"]

for hit in hits:
    print(f"Index: {hit['_index']}, ID: {hit['_id']}, Score: {hit['_score']}")

Inspecting Shard Execution in the Response

The dictionary returned by client.search() contains a _shards key that provides metadata about the distributed execution:

Evaluating response["_shards"] ensures that all nodes and shards contributed to the result without partial failures.