ism index patterns based on indexAlias/index #5117 #5118
Conversation
Signed-off-by: kranthikirang <kranthi.guttikonda9@gmail.com>
kranthikirang
requested review from
sb2k16,
chenqi0805,
engechas,
graytaylor0,
dinujoh,
kkondaka,
KarstenSchnitter,
dlvenable and
oeyh
as code owners
|
Can anyone look into this PR please? @dlvenable |
There was a problem hiding this comment.
Thank you @kranthikirang for this contribution. I have a few questions and then some specific comments.
What was the ISM policy this created for your index?
Did you try with any variables that are not at the end?
...main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/AbstractIndexManager.java
Show resolved
Hide resolved
...va/org/opensearch/dataprepper/plugins/sink/opensearch/index/IsmPolicyManagementStrategy.java
Show resolved
Hide resolved
.../main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IsmPolicyManagement.java
Show resolved
Hide resolved
Signed-off-by: kranthikirang <kranthi.guttikonda9@gmail.com>
Signed-off-by: kranthikirang <kranthi.guttikonda9@gmail.com>
Signed-off-by: kranthikirang <kranthi.guttikonda9@gmail.com>
|
@dlvenable I have added getIndexPattern test case as pert of integration test suite. I have followed the below steps. `docker run --name=opendistro -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -d amazon/opendistro-for-elasticsearch:1.13.3 ./gradlew :data-prepper-plugins:opensearch:integrationTest --tests "org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchIT" -Dtests.opensearch.host=localhost:9200 -Dtests.opensearch.user=admin -Dtests.opensearch.password=**** No issues were observed. Please take a look. |
|
@dlvenable Can you please run the tests again? |
Description
We will be able to update the index_patterns of the ISM policy based on the indexAlias created (more useful when its dynamic and/or with rollover feature) when user supplies ism_policy_file. This will not interfere when user wants to manage the policy with superset of wildcard (which should be a manual step). This commit will only ensure policy's index_patterns are updated when user supplies policy_json_file.
Example:
route:
- cluster-utils: '/route_name == "cluster"'
sink:
hosts: [ "https://opensearch-node1:9200","https://opensearch-node2:9200" ]
Change to your credentials
username: "admin"password: "PGjGcHKfQW1U"
insecure: true
dlq_file: /usr/share/data-prepper/log/data-prepper/dlq.log
Add a certificate file if you are accessing an OpenSearch cluster with a self-signed certificate
#cert: /path/to/certnormalize_index: true
index: cluster-utils_eastus2_${cluster_name}
index_type: custom
template_type: v1
template_file: /usr/share/data-prepper/templates/template-file.json
ism_policy_file: /usr/share/data-prepper/policies/ism-policy.json
routes: [cluster-utils]
Where using data prepper we will create ism policy (if user supplies wildcard cluster-utils*) each time we receive a different cluster_name and will fail because opensearch will not allow having same patterns in multiple policies. Also keeping a single policy wouldn't make sense if we want to manage via AP. with this change data-prepper will create new policy with index pattern as cluster-utils_eastus2_${cluster_name}* which will in-align to index template index_patterns. Also, previous functionality will also work as index_patterns are specific to the indexAlias. However users will have to re-run the policies on the older indexes created.
Extremely useful with rollover through ISM policy.
Issues Resolved
Resolves #5117
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.