Allow status anonymous for cloud tests

[liza-mae]

We are seeing this error on some of the tests in cloud, see: #38594
Although I was also able to reproduce on-prem.

error [16:19:26.356] [error][authentication] [strict_dynamic_mapping_exception] mapping set to strict, dynamic introduction of [references] within [_doc] is not allowed :: {"path":"/.kibana/_create/config%3A7.2.0","query":{"refresh":"wait_for"},"body":"{"config":{"buildNum":24319},"type":"config","references":[],"updated_at":"2019-06-20T16:19:26.343Z"}","statusCode":400,"response":"{"error":{"root_cause":[{"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [references] within [_doc] is not allowed"}],"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [references] within [_doc] is not allowed"},"status":400}"}
at respond (/tmp/kibana-7.2.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:315:15)
at checkRespForFailure (/tmp/kibana-7.2.0-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:274:7)
at HttpConnector. (/tmp/kibana-7.2.0-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)
at IncomingMessage.wrapper (/tmp/kibana-7.2.0-linux-x86_64/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)
at IncomingMessage.emit (events.js:194:15)
at endReadableNT (_stream_readable.js:1103:12)
at process._tickCallback (internal/process/next_tick.js:63:19)

I narrowed it down to requiring to have this setting:
status.allowAnonymous: true

Although I can do this on-prem, but is this something we can/should set on cloud?
Should it work without this setting?
Is there something else missing about this error?

[elasticmachine]

Pinging @elastic/kibana-operations

[elasticmachine]

Pinging @elastic/kibana-security

[elasticmachine]

Pinging @elastic/kibana-test-triage

[liza-mae]

Additional test failures associated with this in x-pack tests canvas, dashboard, discover, rollup, timelion, visualize, saved objects, reporting.

Related to #35061 ?

[liza-mae]

I did try to set this on cloud and it did not appear to work as some of the settings are not applicable on cloud. If setting it to try and being able to access the status page without authentication should work, I am getting: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}

[kobelb]

Cloud only white-lists some settings to work in cloud, I'm not seeing "status.allowAnonymous" listed there. We'll have to work with them to get this setting added.

[liza-mae]

Spoke with Spencer seems like maybe this setting is masking a bug with esArchiver -- he is making a fix to esArchiver-- which we tested locally and it seemed to fix this where it works with and without this setting. I will test his PR against cloud to verify.

[liza-mae]

Just to add some stats: only roughly 27 tests out of 685 have this error with esArchiver and it is not a software/migration issue as was reported to me and others.

[liza-mae]

PR: #39481

[LeeDr]

@liza-mae could you briefly describe how you narrowed the problem down to;
status.allowAnonymous: true ? Was it from Kibana log files run locally? Or access to Kibana logs on Cloud?

[liza-mae]

I narrowed down the problem since it did not make sense to me that a cloud test for canvas smoke would uncover an upgrade / migration issue for 7.2. There had been no other indication that this test or any other test failed anywhere related to the upgrade/migration problem sent to me.

Also the issue that was referenced to me that described a similar output was when upgrading from 6.7 to 7.0 and 6.7.2 to 7.1.1. Hopefully we tested 6.8 to 7.2. I just went to kibana-ci captured the test passing and worked my way to narrow down to the setting.

My focus is to get the tests on cloud running in a stable way, so this failure on canvas led to esArchiver root cause. Testing with esArchiver fixes those tests now pass.

[liza-mae]

Fixed by: #39481