[saved objects] Add migrations v2 integration test for scenario with multiple Kibana instances. #100171

lukeelmers · 2021-05-14T21:12:23Z

This adds integration tests for two scenarios:

Three Kibana instances running migrations at once.
(Confirms v2 algorithm operations are idempotent)
Three Kibana instances with different plugins running migrations at once.
(Confirms there are no possible race conditions between instances that have & don't have certain plugins)

Multiple Kibana instances are simulated by creating multiple Roots. To allow ample time for the possibility of races, the data set is 5k objects, and I'm using a batch size of 100.

This was my first time really diving into the v2 migrations code, maybe others who have worked with it more will have ideas of other ways to try to simulate race conditions 🙂

…multiple Kibana nodes.

elasticmachine · 2021-05-14T21:12:25Z

Pinging @elastic/kibana-core (Team:Core)

src/core/server/saved_objects/migrationsv2/integration_tests/multiple_kibana_nodes.test.ts

lukeelmers · 2021-05-31T16:46:46Z

@elasticmachine merge upstream

lukeelmers · 2021-06-01T16:37:07Z

@elasticmachine merge upstream

kibanamachine · 2021-06-01T16:59:28Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 469a6af
Storybooks Preview
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures

Failed CI Steps

Jest Integration Tests

Test Failures

Kibana Pipeline / general / Jest Integration Tests.src/core/server/saved_objects/migrationsv2/integration_tests.migration v2 migrates saved objects normally when multiple Kibana instances are running

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: Unable to complete saved object migrations for the [.kibana] index. Error: [{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}}][Error: [{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}},{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}}]
    at /dev/shm/workspace/parallel/3/kibana/src/core/server/saved_objects/migrationsv2/actions/index.ts:1232:15
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at stateActionMachine (/dev/shm/workspace/parallel/3/kibana/src/core/server/saved_objects/migrationsv2/state_action_machine.ts:70:28)
    at migrationStateActionMachine (/dev/shm/workspace/parallel/3/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.ts:126:24)
    at async Promise.all (index 0)
    at SavedObjectsService.start (/dev/shm/workspace/parallel/3/kibana/src/core/server/saved_objects/saved_objects_service.ts:412:7)
    at Server.start (/dev/shm/workspace/parallel/3/kibana/src/core/server/server.ts:235:31)
    at Root.start (/dev/shm/workspace/parallel/3/kibana/src/core/server/root/index.ts:52:14)
    at async Promise.all (index 0)
    at Object.<anonymous> (/dev/shm/workspace/parallel/3/kibana/src/core/server/saved_objects/migrationsv2/integration_tests/multiple_kibana_nodes.test.ts:179:5)]

Metrics [docs]

Unknown metric groups

References to deprecated APIs

id	before	after	diff
`canvas`	29	25	-4
`crossClusterReplication`	8	6	-2
`fleet`	22	20	-2
`globalSearch`	4	2	-2
`indexManagement`	12	7	-5
`infra`	256	146	-110
`lens`	67	45	-22
`licensing`	18	15	-3
`lists`	239	236	-3
`maps`	286	208	-78
`ml`	121	115	-6
`monitoring`	109	56	-53
`securitySolution`	390	346	-44
`stackAlerts`	101	95	-6
total			-340

History

💔 Build #128359 failed 6e3675f
💚 Build #128229 succeeded 16f732f
💚 Build #126114 succeeded 9c24603

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @lukeelmers

lukeelmers · 2021-06-02T22:51:57Z

Error: Unable to complete saved object migrations for the [.kibana] index. Error: [{"index":{"error":{"type":"cluster_block_exception","reason":"index [.kibana_8.0.0_reindex_temp] blocked by: [FORBIDDEN/8/moving to block index write (api)];"}}}

I hadn't seen these failures until this week (perhaps just a coincidence), so I ran it through the flaky test runner overnight, and it failed 30 out of 100 times 🙁

As a comparison, I ran this same branch through the flaky test runner with the v1 migrations enabled, and all 100 runs passed (makes sense I guess, as we weren't using write blocks in the old algorithm).

So I did some digging. AFAICT v2 migrations aren't handling this particular type of cluster_block_exception -- the only specific error handling around index write failures is in the regex here, which doesn't seem to handle moving to block index write.

kibana/src/core/server/saved_objects/migrationsv2/actions/index.ts

Lines 743 to 745 in d6828a2

    
           const failureIsAWriteBlock = ({ cause: { type, reason } }: WaitForReindexTaskFailure) => 
        
             type === 'cluster_block_exception' && 
        
             reason.match(/index \[.+] blocked by: \[FORBIDDEN\/8\/index write \(api\)\]/);

The error in the stack trace comes from bulkOverwriteTransformedDocuments (which seems to be called later in REINDEX_SOURCE_TO_TEMP_INDEX_BULK):

kibana/src/core/server/saved_objects/migrationsv2/actions/index.ts

Lines 1229 to 1233 in d6828a2

    
           if (errors.length === 0) { 
        
             return Either.right('bulk_index_succeeded' as const); 
        
           } else { 
        
             throw new Error(JSON.stringify(errors)); 
        
           }

I'm not super familiar with this API, but seems it's coming from here in ES:
https://github.com/elastic/elasticsearch/blob/5baabff6670a8ed49297488ca8cac8ec12a2078d/server/src/main/java/org/elasticsearch/cluster/metadata/MetadataIndexStateService.java#L929-L935

Perhaps it's a scenario we didn't consider (or run into), where multiple Kibanas migrating in parallel could also hit this error, if one starts creating the write block before the others? When I stuck in some code here to ignore any moving to block index write errors and continue with the next step, tests started passing again.

Would appreciate any thoughts @elastic/kibana-core -- should we be handling this error in v2 migrations, or am I doing something unusual in the test that would trigger this error in a way would not realistically happen for users?

pgayvallet · 2021-06-03T09:28:32Z

FORBIDDEN/8/moving to block index write (api)

This is happening because we're setting write block to true for the temp index during the SET_TEMP_WRITE_BLOCK stage, right between REINDEX_SOURCE_TO_TEMP_CLOSE_PIT and CLONE_TEMP_TO_TARGET

So, yea, in a situation where instance1 is already at CLONE_TEMP_TO_TARGET or further, and instance2 enters the REINDEX_SOURCE_TO_TEMP_INDEX_BULK stage, it will encounter this error and fail.

This is surprising you didn't encounter this issue sooner while implementing these tests btw, as it seems a relatively 'easy' race condition to reproduce if both instances are not started at the exact same time.

Now, for the resolution, it kinda depends on what we will choose to do regarding #101052

If we decide to keep the existing behavior and have the migration fails if the instance encounters unknown SO types, I think it would be acceptable to have instance2 just jump from REINDEX_SOURCE_TO_TEMP_INDEX_BULK to somewhere near CLONE_TEMP_TO_TARGET if this error is encountered, as I think we can assume that one other instance already successfully completed the source->temp reindex (else the write block would not be present). Note that we still need to also decide of the behavior when multiple instances have different plugins set, as this would lead to inconsistencies.

Now, if we decide to copy the unknown documents instead (and allow instances to have different set of plugins enabled), this can be way trickier, as I thinkwe can't safely perform the same step jump? (also see #101052 (comment))

lukeelmers · 2021-06-04T18:42:00Z

Putting this PR on hold pending the outcome of #101351

Once that's resolved we should update/merge this, or incorporate it into that PR.

For the time being, I'll move this back into draft.

pgayvallet · 2021-07-07T08:17:23Z

Closing in favor of #104516

[saved objects] Add migrations v2 integration test for scenario with …

9c24603

…multiple Kibana nodes.

lukeelmers requested a review from a team as a code owner May 14, 2021 21:12

lukeelmers commented May 14, 2021

View reviewed changes

mshustov reviewed May 26, 2021

View reviewed changes

joshdover reviewed May 26, 2021

View reviewed changes

joshdover mentioned this pull request May 27, 2021

Minimize downtime while remedying corrupt document migration failures #100768

Open

lukeelmers self-assigned this May 31, 2021

Clean up tests based on feedback.

8ec2352

Merge branch 'master' into tests-sov2

16f732f

joshdover mentioned this pull request Jun 1, 2021

Fail v2 Migrations when unknown document types are found #101052

Closed

Remove second test pending outcome of elastic#101052.

6e3675f

Merge branch 'master' into tests-sov2

469a6af

lukeelmers removed auto-backport Deprecated - use backport:version if exact versions are needed v7.12.2 v7.13.1 labels Jun 4, 2021

lukeelmers marked this pull request as draft June 4, 2021 18:42

lukeelmers mentioned this pull request Jun 4, 2021

SO migration v2: fail the migration early if unknown types are encountered #101351

Closed

mshustov marked this pull request as ready for review June 5, 2021 08:04

mshustov marked this pull request as draft June 5, 2021 08:04

spalger added v7.15.0 and removed v7.14.0 labels Jun 30, 2021

pgayvallet mentioned this pull request Jul 6, 2021

[SO Migration] fix reindex race on multi-instance mode #104516

Merged

2 tasks

pgayvallet closed this Jul 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[saved objects] Add migrations v2 integration test for scenario with multiple Kibana instances. #100171

[saved objects] Add migrations v2 integration test for scenario with multiple Kibana instances. #100171

lukeelmers commented May 14, 2021

elasticmachine commented May 14, 2021

lukeelmers commented May 31, 2021

lukeelmers commented Jun 1, 2021

kibanamachine commented Jun 1, 2021 •

edited

Loading

Standard Out

Stack Trace

References to deprecated APIs

lukeelmers commented Jun 2, 2021

pgayvallet commented Jun 3, 2021

lukeelmers commented Jun 4, 2021

pgayvallet commented Jul 7, 2021

[saved objects] Add migrations v2 integration test for scenario with multiple Kibana instances. #100171

[saved objects] Add migrations v2 integration test for scenario with multiple Kibana instances. #100171

Conversation

lukeelmers commented May 14, 2021

elasticmachine commented May 14, 2021

lukeelmers commented May 31, 2021

lukeelmers commented Jun 1, 2021

kibanamachine commented Jun 1, 2021 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Standard Out

Stack Trace

Metrics [docs]

References to deprecated APIs

History

lukeelmers commented Jun 2, 2021

pgayvallet commented Jun 3, 2021

lukeelmers commented Jun 4, 2021

pgayvallet commented Jul 7, 2021

kibanamachine commented Jun 1, 2021 •

edited

Loading