Amazon OpenSearch Service 101: What number of shards do I want

Prospects new to Amazon OpenSearch Service typically ask what number of shards their indexes want. An index is a set of shards, and an index’s shard depend can have an effect on each indexing and search request effectivity. OpenSearch Service can soak up massive quantities of information, break up it into smaller models known as shards, and distribute these shards throughout a dynamically altering set of situations.

On this put up, we offer some sensible steering for figuring out the best shard depend to your use case.

Shards overview

A search engine has two jobs: create an index from a set of paperwork, and search that index to compute the best-matching paperwork. In case your index is sufficiently small, a single partition on a single machine can retailer that index. For bigger doc units, in instances the place a single machine isn’t massive sufficient to carry the index, or in instances the place a single machine can’t compute your search outcomes successfully, the index might be break up into partitions. These partitions are known as shards in OpenSearch Service. Every doc is routed to a shard that’s calculated, by default, by utilizing a hash of that doc’s ID.

A shard is each a unit of storage and a unit of computation. OpenSearch Service distributes shards throughout nodes in your cluster to parallelize index storage and processing. For those who add extra nodes to an OpenSearch Service area, it mechanically rebalances the shards by transferring them between the nodes. The next determine illustrates this course of.

As storage, major shards are distinct from each other. The doc set in a single shard doesn’t overlap the doc set in different shards. This strategy makes shards unbiased for storage.

As computational models, shards are additionally distinct from each other. Every shard is an occasion of an Apache Lucene index that computes outcomes on the paperwork it holds. As a result of all of the shards comprise the index, they need to operate collectively to course of every question and replace request for that index. To course of a question, OpenSearch Service routes the question to a knowledge node for a major or duplicate shard. Every node computes its response domestically and the shard responses get aggregated for a ultimate response. To course of a write request (a doc ingestion or an replace to an present doc), OpenSearch Service routes the request to the suitable shards—major then duplicate. As a result of most writes are bulk requests, all shards of an index are sometimes used.

The 2 various kinds of shards

There are two sorts of shards in OpenSearch Service—major and duplicate shards. In an OpenSearch index configuration, the first shard depend serves to partition information and the duplicate depend is the variety of full copies of the first shards. For instance, in the event you configure your index with 5 major shards and 1 duplicate, you’ll have a complete of 10 shards: 5 major shards and 5 duplicate shards.

The first shard receives writes first. The first shard passes paperwork to the duplicate shards for indexing by default. OpenSearch Service’s O-series situations use phase replication. By default, OpenSearch Service waits for acknowledgment from duplicate shards earlier than confirming a profitable write operation to the shopper. Major and duplicate shards present redundant information storage, enhancing cluster resilience in opposition to node failures. Within the following instance, the OpenSearch Service area has three information nodes. There are two indexes, inexperienced (darker) and blue (lighter), every of which has three shards. The first for every shard is printed in crimson. Every shard additionally has a single duplicate, proven with no define.

OpenSearch Service maps shards to nodes based mostly on various guidelines. Probably the most primary rule is that major and duplicate shards are by no means put onto the identical node. If an information node fails, OpenSearch Service mechanically creates one other information node and re-replicates shards from surviving nodes and redistributes them throughout the cluster. If major shards fail, duplicate shards are promoted to major to stop information loss and supply steady indexing and search operations.

So what number of shards? Deal with storage first

There are three kinds of workloads that OpenSearch customers sometimes preserve: seek for functions, log analytics, and as a vector database. Search workloads are read-heavy and latency delicate. They’re sometimes tied to an software to boost search functionality and efficiency. A typical sample is to index the info in relational databases to provide customers extra filtering capabilities and supply environment friendly full textual content search.

Log workloads are write-heavy and obtain information constantly from functions and community gadgets. Usually, that information is put right into a altering set of indexes, based mostly on an indexing time interval like day by day or month-to-month relying on the use case. As an alternative of indexing based mostly on time interval, you should utilize rollover insurance policies based mostly on index measurement or doc depend to ensure shard sizing greatest practices are adopted.

Vector database workloads use the OpenSearch Service k-Nearest Neighbor (k-NN) plugin to index vectors from an embedding pipeline. This permits semantic search, which measures relevance utilizing the which means of phrases moderately than precisely matching the phrases. The embedding mannequin from the pipeline maps multimodal information right into a vector with probably hundreds of dimensions. OpenSearch Service searches throughout vectors to supply search outcomes.

To find out the optimum variety of shards to your workload, begin together with your index storage necessities. Though storage necessities can fluctuate broadly, a common guideline is to make use of 1:1.25 utilizing the supply information measurement to estimate utilization. Additionally, compression algorithms default to efficiency, however will also be adjusted to scale back measurement. In terms of shard sizes, think about the next based mostly on the workload:

• Search – Divide your complete storage requirement by 30 GB.
  • If search latency is excessive, use a smaller shard measurement (as little as 10GB), rising the shard depend and parallelism for question processing.
  • Growing the shard depend reduces the quantity of labor at every shard (they’ve fewer paperwork to course of), but in addition will increase the quantity of networking for distributing the question and gathering the response. To stability these competing issues, look at your common hit depend. In case your hit depend is excessive, use smaller shards. In case your hit depend is low, use bigger shards.
• Logs – Divide the storage requirement to your desired time interval by 50 GB.
  • If utilizing an ISM coverage with rollover, think about setting the min_size parameter to 50 GB.
  • Growing the shard depend for logs workloads equally improves parallelism. Nonetheless, most queries for logs workloads have a small hit depend, so question processing is mild. Logs workloads work effectively with bigger shard sizes, however shard smaller in case your question workload is heavier.
• Vector – Divide your complete storage requirement by 50 GB.
  • Decreasing shard measurement (as little as 10GB) can enhance search latency when your vector queries are hybrid with a heavy lexical part. Conversely, rising shard measurement (as excessive as 75GB) can enhance latency when your queries are pure vector queries.
  • OpenSearch offers different optimization strategies for vector databases, together with vector quantization and disk-based search.
  • Okay-NN queries behave like extremely filtered search queries, with low hit counts. Due to this fact, bigger shards are inclined to work effectively. Be ready to shard smaller when your queries are heavier.

Don’t be afraid of utilizing a single shard

In case your index comprises lower than the suggested shard measurement (30 GB for search and 50 GB in any other case), we advocate that you simply use a single major shard. Though it’s tempting so as to add extra shards considering it can enhance efficiency, this strategy can really be counterproductive for smaller datasets due to the added networking. Every shard you add to an index distributes the processing of requests for that index throughout an extra node. Efficiency can lower as a result of there may be overhead for distributed operations to separate and mix outcomes throughout nodes when a single node can do it sufficiently.

Set the shard depend

While you create an OpenSearch index, you set the first and duplicate counts for that index. As a result of you possibly can’t dynamically change the first shard depend of an present index, it’s important to make this vital configuration choice earlier than indexing your first doc.

You set the shard depend utilizing the OpenSearch create index API. For instance (present your OpenSearch Service area endpoint URL and index identify):

curl -XPUT https:/// -H 'Content material-Kind: software/json' -d 
 '{
    "settings": {
        "index" : {
            "number_of_shards": 3,
            "number_of_replicas": 1
        }
    }
 }'

In case you have a single index workload, you solely have to do that one time, whenever you create your index for the primary time. In case you have a rolling index workload, you create a brand new index commonly. Use the index template API to automate making use of settings to all new indexes whose identify matches the template. The next instance units the shard depend for any index whose identify has the prefix logs (present your OpenSearch service endpoint area URL and index template identify):

curl -XPUT https:///_index_template/ -H 'Content material-Kind: software/json' -d 
 '{
   "index_patterns": ["logs*"],
   "template": {
        "settings": {
            "index" : {
                "number_of_shards": 3,
                "number_of_replicas": 1
            }
       }
  }
}'

Conclusion

This put up outlined primary shard sizing greatest practices, however extra elements would possibly affect the best index configuration you select to implement in your OpenSearch Service area.

For extra details about sharding, confer with Optimize OpenSearch index shard sizes or Shard technique. Each sources can assist you higher fine-tune your OpenSearch Service area to optimize its out there compute sources.

In regards to the authors

Tom Burns is a Senior Cloud Assist Engineer at AWS and is predicated within the NYC space. He’s an issue professional in Amazon OpenSearch Service and engages with clients for vital occasion troubleshooting and enhancing the supportability of the service. Outdoors of labor, he enjoys taking part in together with his cats, taking part in board video games with buddies, and taking part in aggressive video games on-line.

Ron Miller is a Options Architect based mostly out of NYC, supporting transportation and logistics clients. Ron works carefully with AWS’s Information & Analytics specialist group to advertise and assist OpenSearch. On the weekend, Ron is a shade tree mechanic and trains to finish triathlons.