From bf2058134a9cecc6888b4d17318edb833c4f9366 Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Sun, 7 Aug 2022 21:14:17 -0400 Subject: [PATCH 1/3] Added replica count enforcement Signed-off-by: Fanit Kolchina --- _opensearch/cluster.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/_opensearch/cluster.md b/_opensearch/cluster.md index 22ac9c8c16..a9d0c365f3 100644 --- a/_opensearch/cluster.md +++ b/_opensearch/cluster.md @@ -176,6 +176,8 @@ To better understand and monitor your cluster, use the [cat API]({{site.url}}{{s ## (Advanced) Step 6: Configure shard allocation awareness or forced awareness +### Shard allocation awareness + If your nodes are spread across several geographical zones, you can configure shard allocation awareness to allocate all replica shards to a zone that’s different from their primary shard. With shard allocation awareness, if the nodes in one of your zones fail, you can be assured that your replica shards are spread across your other zones. It adds a layer of fault tolerance to ensure your data survives a zone failure beyond just individual node failures. @@ -205,6 +207,8 @@ You can either use `persistent` or `transient` settings. We recommend the `persi Shard allocation awareness attempts to separate primary and replica shards across multiple zones. However, if only one zone is available (such as after a zone failure), OpenSearch allocates replica shards to the only remaining zone. +### Forced awareness + Another option is to require that primary and replica shards are never allocated to the same zone. This is called forced awareness. To configure forced awareness, specify all the possible values for your zone attributes: @@ -226,6 +230,28 @@ If that is not the case, and `opensearch-d1` and `opensearch-d2` do not have the Choosing allocation awareness or forced awareness depends on how much space you might need in each zone to balance your primary and replica shards. +### Replica count enforcement + +To enforce an even distribution of shards across all zones and avoid hotspots, you can set the `routing.allocation.awareness.balance` attribute to `true`. This setting can be configured in the opensearch.yml file and dynamically updated using the cluster update settings API: + +```json +PUT _cluster/settings +{ + "persistent": { + "cluster": { + "routing.allocation.awareness.balance": "true" + } + } +} +``` + +The `routing.allocation.awareness.balance` setting is false by default. When it is set to `true`, the total number of shards for the index must be a multiple of the maximum value of all awareness attribute counts. For example, consider a configuration with two awareness attributes set up — zones and rack IDs. Let's say there are two zones and three rack IDs. The maximum of the number of zones and the number of rack IDs is three. Therefore, the number of shards must be a multiple of three. If it is not, OpenSearch throws a validation exception. + +`routing.allocation.awareness.balance` takes effect only if `cluster.routing.allocation.awareness.attributes` and `cluster.routing.allocation.awareness.force.zone.values` are set. +{: .note} + +`routing.allocation.awareness.balance` applies to all operations that create or update indices. For example, let's say you're running a cluster with three nodes and three zones, in a zone-aware setting. If you try to create an index with one replica, or update an index's settings to one replica, the attempt will fail with a validation exception because the number of shards must be a multiple of three. Similarly, if you try to create an index template with one shard and no replicas, the attempt will fail for the same reason. However, in all of those operations, if you set the number of shards to one and the number of replicas to two, the total number of shards is three and the attempt will succeed. + ## (Advanced) Step 7: Set up a hot-warm architecture You can design a hot-warm architecture where you first index your data to hot nodes---fast and expensive---and after a certain period of time move them to warm nodes---slow and cheap. From 13eb060cb5562183ca5098602b4e848fe978b06e Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Mon, 8 Aug 2022 13:07:27 -0400 Subject: [PATCH 2/3] Incorporated review comments Signed-off-by: Fanit Kolchina --- _opensearch/cluster.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_opensearch/cluster.md b/_opensearch/cluster.md index a9d0c365f3..ce2c62b0f6 100644 --- a/_opensearch/cluster.md +++ b/_opensearch/cluster.md @@ -245,7 +245,7 @@ PUT _cluster/settings } ``` -The `routing.allocation.awareness.balance` setting is false by default. When it is set to `true`, the total number of shards for the index must be a multiple of the maximum value of all awareness attribute counts. For example, consider a configuration with two awareness attributes set up — zones and rack IDs. Let's say there are two zones and three rack IDs. The maximum of the number of zones and the number of rack IDs is three. Therefore, the number of shards must be a multiple of three. If it is not, OpenSearch throws a validation exception. +The `routing.allocation.awareness.balance` setting is false by default. When it is set to `true`, the total number of shards for the index must be a multiple of the highest count for any awareness attribute. For example, consider a configuration with two awareness attributes set up — zones and rack IDs. Let's say there are two zones and three rack IDs. The highest count of either the number of zones or the number of rack IDs is three. Therefore, the number of shards must be a multiple of three. If it is not, OpenSearch throws a validation exception. `routing.allocation.awareness.balance` takes effect only if `cluster.routing.allocation.awareness.attributes` and `cluster.routing.allocation.awareness.force.zone.values` are set. {: .note} From 3a6bf7dfe5381239934278eca47006879e632d0c Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Mon, 8 Aug 2022 16:23:51 -0400 Subject: [PATCH 3/3] Incorporated editorial comments Signed-off-by: Fanit Kolchina --- _opensearch/cluster.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_opensearch/cluster.md b/_opensearch/cluster.md index ce2c62b0f6..c4a2bd052d 100644 --- a/_opensearch/cluster.md +++ b/_opensearch/cluster.md @@ -245,12 +245,12 @@ PUT _cluster/settings } ``` -The `routing.allocation.awareness.balance` setting is false by default. When it is set to `true`, the total number of shards for the index must be a multiple of the highest count for any awareness attribute. For example, consider a configuration with two awareness attributes set up — zones and rack IDs. Let's say there are two zones and three rack IDs. The highest count of either the number of zones or the number of rack IDs is three. Therefore, the number of shards must be a multiple of three. If it is not, OpenSearch throws a validation exception. +The `routing.allocation.awareness.balance` setting is false by default. When it is set to `true`, the total number of shards for the index must be a multiple of the highest count for any awareness attribute. For example, consider a configuration with two awareness attributes—zones and rack IDs. Let's say there are two zones and three rack IDs. The highest count of either the number of zones or the number of rack IDs is three. Therefore, the number of shards must be a multiple of three. If it is not, OpenSearch throws a validation exception. `routing.allocation.awareness.balance` takes effect only if `cluster.routing.allocation.awareness.attributes` and `cluster.routing.allocation.awareness.force.zone.values` are set. {: .note} -`routing.allocation.awareness.balance` applies to all operations that create or update indices. For example, let's say you're running a cluster with three nodes and three zones, in a zone-aware setting. If you try to create an index with one replica, or update an index's settings to one replica, the attempt will fail with a validation exception because the number of shards must be a multiple of three. Similarly, if you try to create an index template with one shard and no replicas, the attempt will fail for the same reason. However, in all of those operations, if you set the number of shards to one and the number of replicas to two, the total number of shards is three and the attempt will succeed. +`routing.allocation.awareness.balance` applies to all operations that create or update indices. For example, let's say you're running a cluster with three nodes and three zones in a zone-aware setting. If you try to create an index with one replica or update an index's settings to one replica, the attempt will fail with a validation exception because the number of shards must be a multiple of three. Similarly, if you try to create an index template with one shard and no replicas, the attempt will fail for the same reason. However, in all of those operations, if you set the number of shards to one and the number of replicas to two, the total number of shards is three and the attempt will succeed. ## (Advanced) Step 7: Set up a hot-warm architecture