[BUG] org.opensearch.client.indices.GetIndexTemplatesResponseTests.testParsingFromOpenSearchResponse is flaky

[reta]

Describe the bug

OpenSearchParseException[Failed to parse content to map]; nested: JsonParseException[Duplicate field 'CZoEK'
 at [Source: (byte[])[247 bytes]; byte offset: #101]];
	at __randomizedtesting.SeedInfo.seed([3D0DAE5BAD265A1F:76B30BB90A00AF61]:0)
	at app//org.opensearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:258)
	at app//org.opensearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:185)
	at app//org.opensearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:156)
	at app//org.opensearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:141)
	at app//org.opensearch.cluster.metadata.IndexTemplateMetadata$Builder.toInnerXContent(IndexTemplateMetadata.java:425)
	at app//org.opensearch.cluster.metadata.IndexTemplateMetadata$Builder.toXContent(IndexTemplateMetadata.java:397)
	at app//org.opensearch.action.admin.indices.template.get.GetIndexTemplatesResponse.toXContent(GetIndexTemplatesResponse.java:102)
	at app//org.opensearch.client.indices.GetIndexTemplatesResponseTests.testParsingFromOpenSearchResponse(GetIndexTemplatesResponseTests.java:111)
	at java.base@19.0.2/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)

To Reproduce
See please https://build.ci.opensearch.org/job/gradle-check/12469/testReport/junit/org.opensearch.client.indices/GetIndexTemplatesResponseTests/testParsingFromOpenSearchResponse/

Expected behavior
The test should always pass

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context
Add any other context about the problem here.

[dbwiddis]

The fields in the metadata being parsed in this test are randomly generated using randomAlphaOfLength(5). Given 52 possible characters there is a 1 in 380,204,032 chance of a collision between any pair. But there are up to 32 such indices created, so we have a Birthday Problem with 32*31=992 pairs that can potentially collide. This increases the probability to 1 in 383,271. But the test is run 20 times, so the probability the test will fail is 1 in 19,164.

The test failed on run 12,469, which seems in line with this expectation. We'll probably get another failure in a few years.

To me this seems like an acceptable level of flakiness, and if this failure has only been seen once I'm fine with closing this issue as a wontfix. Other options:

• retry the test on failure once.
• keep track of our randomly generated alpha keys to avoid these rare duplicates
• add some randomness to the key length so they're not all length 5, to make it even more rare

[dbwiddis]

No other occurrences of this failure in previous 2500 runs, closed as "not flaky enough"

[andrross]

keep track of our randomly generated alpha keys to avoid these rare duplicates

@dbwiddis Is this fix as simple as preventing duplicate keys in this helper method? If so, I'd be inclined to do that fix.

[dbwiddis]

@dbwiddis Is this fix as simple as preventing duplicate keys in this helper method? If so, I'd be inclined to do that fix.

Yes, for this specific failure. I also think there's a potential collision with the alias metadata in this line.

I'll go ahead and submit the fix(es).