testCountAllIndicesNoQuery fails

[DaveCTurner]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.0+multijob-unix-compatibility/os=ubuntu-18.04&&immutable/56/console

Here is the REPRODUCE WITH line:

REPRODUCE WITH: ./gradlew :client:rest-high-level:integTestRunner \
  -Dtests.seed=6C5D5F5D2AF87ABE \
  -Dtests.class=org.elasticsearch.client.SearchIT \
  -Dtests.method="testCountAllIndicesNoQuery" \
  -Dtests.security.manager=true \
  -Dtests.locale=lt \
  -Dtests.timezone=Brazil/East \
  -Dcompiler.java=11 \
  -Druntime.java=8

However this doesn't reproduce for me. There's not a lot in the logs, but here it is:

Suite: org.elasticsearch.client.SearchIT
  1> [2019-03-06T07:27:59,138][INFO ][o.e.c.SearchIT           ] [testCountAllIndicesNoQuery] before test
  1> [2019-03-06T07:27:59,140][INFO ][o.e.c.SearchIT           ] [testCountAllIndicesNoQuery] initializing REST clients against [http://[::1]:45017]
  2> REPRODUCE WITH: ./gradlew :client:rest-high-level:integTestRunner -Dtests.seed=6C5D5F5D2AF87ABE -Dtests.class=org.elasticsearch.client.SearchIT -Dtests.method="testCountAllIndicesNoQuery" -Dtests.security.manager=true -Dtests.locale=lt -Dtests.timezone=Brazil/East -Dcompiler.java=11 -Druntime.java=8
  1> [2019-03-06T07:27:59,286][WARN ][o.e.c.RestClient         ] [testCountAllIndicesNoQuery] request [PUT http://[::1]:45017/index/type/1] returned 1 warnings: [299 Elasticsearch-7.0.0-SNAPSHOT-65b44ee "[types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id})."]
  1> [2019-03-06T07:27:59,295][WARN ][o.e.c.RestClient         ] [testCountAllIndicesNoQuery] request [PUT http://[::1]:45017/index/type/2] returned 1 warnings: [299 Elasticsearch-7.0.0-SNAPSHOT-65b44ee "[types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id})."]
  1> [2019-03-06T07:27:59,304][WARN ][o.e.c.RestClient         ] [testCountAllIndicesNoQuery] request [PUT http://[::1]:45017/index/type/3] returned 1 warnings: [299 Elasticsearch-7.0.0-SNAPSHOT-65b44ee "[types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id})."]
  1> [2019-03-06T07:27:59,312][WARN ][o.e.c.RestClient         ] [testCountAllIndicesNoQuery] request [PUT http://[::1]:45017/index/type/4] returned 1 warnings: [299 Elasticsearch-7.0.0-SNAPSHOT-65b44ee "[types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id})."]
  1> [2019-03-06T07:27:59,320][WARN ][o.e.c.RestClient         ] [testCountAllIndicesNoQuery] request [PUT http://[::1]:45017/index/type/5] returned 1 warnings: [299 Elasticsearch-7.0.0-SNAPSHOT-65b44ee "[types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id})."]
  1> [2019-03-06T07:27:59,938][INFO ][o.e.c.SearchIT           ] [testCountAllIndicesNoQuery] Unexpected hit count, was expecting 12 hits but got: {"took":4,"timed_out":false,"_shards":{"total":6,"successful":6,"skipped":0,"failed":0},"hits":{"total":{"value":13,"relation":"eq"},"max_score":1.0,"hits":[{"_index":".tasks","_type":"task","_id":"mOsNh9LZSPu1OI8SS_Znmg:13240","_score":1.0,"_source":{"completed":true,"task":{"node":"mOsNh9LZSPu1OI8SS_Znmg","id":13240,"type":"transport","action":"cluster:admin/xpack/upgrade","description":"","start_time_in_millis":1551868078904,"running_time_in_nanos":354993,"cancellable":true,"headers":{}},"error":{"type":"illegal_state_exception","reason":"Index [test] cannot be upgraded"}}},{"_index":"index","_type":"type","_id":"1","_score":1.0,"_source":{"type":"type1","num":10,"num2":50}},{"_index":"index","_type":"type","_id":"2","_score":1.0,"_source":{"type":"type1","num":20,"num2":40}},{"_index":"index","_type":"type","_id":"3","_score":1.0,"_source":{"type":"type1","num":50,"num2":35}},{"_index":"index","_type":"type","_id":"4","_score":1.0,"_source":{"type":"type2","num":100,"num2":10}},{"_index":"index","_type":"type","_id":"5","_score":1.0,"_source":{"type":"type2","num":100,"num2":10}},{"_index":"index1","_type":"_doc","_id":"1","_score":1.0,"_source":{"field":"value1","rating":7}},{"_index":"index1","_type":"_doc","_id":"2","_score":1.0,"_source":{"field":"value2"}},{"_index":"index2","_type":"_doc","_id":"3","_score":1.0,"_source":{"field":"value1","rating":"good"}},{"_index":"index2","_type":"_doc","_id":"4","_score":1.0,"_source":{"field":"value2"}},{"_index":"index3","_type":"_doc","_id":"5","_score":1.0,"_source":{"field":"value1"}},{"_index":"index3","_type":"_doc","_id":"6","_score":1.0,"_source":{"field":"value2"}},{"_index":"index4","_type":"_doc","_id":"1","_score":1.0,"_source":{"field1":"value1","field2":"value2"}}]}}
  1> [2019-03-06T07:28:00,131][INFO ][o.e.c.SearchIT           ] [testCountAllIndicesNoQuery] after test
FAILURE 1.01s | SearchIT.testCountAllIndicesNoQuery <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: expected:<12> but was:<13>
   > 	at __randomizedtesting.SeedInfo.seed([6C5D5F5D2AF87ABE:9976862BA0CBD654]:0)
   > 	at org.elasticsearch.client.SearchIT.testCountAllIndicesNoQuery(SearchIT.java:1336)
   > 	at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/es-search

[cbuescher]

Maybe unrelated, but I also saaw off-by-one errors in SearchIT recently (#39659). Its wasn't reproducable either but I left it unmuted for now, will see if that one happens again.

Scratch that, different test class. Sorry for the noise.

[jtibshirani]

Looking at the logging statement Unexpected hit count, was expecting 12 hits but got: ... , it seems there is an unexpected result from the tasks index:

    "_id": "mOsNh9LZSPu1OI8SS_Znmg:13240",
    "_index": ".tasks",
    "_score": 1.0,
    "_source": {
        "completed": true,
        "error": {
            "reason": "Index [test] cannot be upgraded",
            "type": "illegal_state_exception"
        },
        "task": {
            "action": "cluster:admin/xpack/upgrade",
            "cancellable": true,
            "description": "",
            "headers": {},
            "id": 13240,
            "node": "mOsNh9LZSPu1OI8SS_Znmg",
            "running_time_in_nanos": 354993,
            "start_time_in_millis": 1551868078904,
            "type": "transport"
        }
    },
    "_type": "task"
}

This looks like a very similar issue to #35644, which we fixed in #37302 by making sure to wait for tasks to terminate in the high-level REST tests. The specific task causing the problem is related to xpack upgrades, and was likely a result of MigrationIT#testUpgradeWithTaskApi.

However I don't think any action is required, since we recently removed this API and associated tests in #40075. I'm going to close out the issue, but we re-open it if a similar failure pops up again.