[CI] open search contexts not cleared in reindex tests on 5.2 branch

[dakrone]

On the 5.2 branch of Elasticsearch CI has seen:

> Throwable #1: java.lang.AssertionError: Failure at [reindex/90_remote:357]: indices.source.total.search.open_contexts didn't match the expected value:
   > indices.source.total.search.open_contexts: expected [0] but was [5]
   > 	at __randomizedtesting.SeedInfo.seed([80A75130B3500549:8F36EEA1DAC68B1]:0)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:374)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:354)
   > 	at java.lang.Thread.run(Thread.java:745)
   > Caused by: java.lang.AssertionError: indices.source.total.search.open_contexts didn't match the expected value:
   > indices.source.total.search.open_contexts: expected [0] but was [5]
   > 	at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:92)
   > 	at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:370)
   > 	... 37 more

I've tried reproducing with

gradle :modules:reindex:integTest -Dtests.seed=80A75130B3500549 -Dtests.class=org.elasticsearch.index.reindex.ReindexClientYamlTestSuiteIT -Dtests.method="test {yaml=reindex/90_remote/Reindex from remote with size}" -Dtests.security.manager=true -Dtests.locale=es-HN -Dtests.timezone=Pacific/Noumea

But it did not reproduce.

Originally seen at: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.2+multijob-unix-compatibility/os=oraclelinux/130/console

[nik9000]

Looks like it could be fixed by #23677 but I didn't want to push that back beyond 5.4 for fear of destabilizing those branches.

[dakrone]

@nik9000 okay, do you think it'd be better to disable the test, or further backport that PR?

[jasontedor]

I think we can set the 5.2 branches to build on push only.

[nik9000]

Since this looks like #23677 I'm ok not fixing it. That was really more of a testing issue than anything. I'd feel bad muting the tests because they are useful. I'd prefer going push only on the branches, but I can certainly backport if we think that is a better choice.

[jasontedor]

I've asked for the 5.2 branches to be set to push only, this is something that we should do anyway.

[jaymode]

Same issue seems to be happening on the 5.3 branch as well. https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.3+multijob-unix-compatibility/os=ubuntu/101/console

Should we mute the test or remove the check in these branches if we're not planning on fixing the issue?

[nik9000]

@jaymode the problem effects all the reindex-from-remote tests. I don't think muting the tests is a great choice. I'm not really sure what to do in situations like these.

[jasontedor]

I think that we should keep them running. These branches are push only, and the failures are seemingly rare. I'd rather there be some coverage than no coverage.

[dakrone]

These branches are push only, and the failures are seemingly rare. I'd rather there be some coverage than no coverage.

I thought 5.3 was still running periodically? We just released it, so it seems like it should be run periodically

[jasontedor]

Okay, I missed that there was a 5.3 dropped here but that doesn't change the fact that some coverage is better than no coverage, and this isn't that noisy.