Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing test: Jest Integration Tests.src/core/server/saved_objects/migrationsv2/actions/integration_tests - migration actions reindex & waitForReindexTask resolves left wait_for_task_completion_timeout when the task does not finish within the timeout #103231

Closed
kibanamachine opened this issue Jun 24, 2021 · 23 comments · Fixed by #113127 or #113382
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Feature:Saved Objects impact:critical This issue should be addressed immediately due to a critical level of impact on the product. Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Jun 24, 2021

A test failed on a tracked branch

Error: expect(received).resolves.toMatchObject(expected)

- Expected  - 6
+ Received  + 1

  Object {
-   "_tag": "Left",
-   "left": Object {
-     "error": Any<ResponseError>,
-     "message": StringMatching /\[timeout_exception\] Timed out waiting for completion of \[org.elasticsearch.index.reindex.BulkByScrollTask/,
-     "type": "wait_for_task_completion_timeout",
-   },
+   "_tag": "Right",
  }
    at Object.toMatchObject (/dev/shm/workspace/parallel/3/kibana/node_modules/expect/build/index.js:198:20)
    at Object.<anonymous> (/dev/shm/workspace/parallel/3/kibana/src/core/server/saved_objects/migrationsv2/actions/integration_tests/actions.test.ts:799:37)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at _callCircusTest (/dev/shm/workspace/parallel/3/kibana/node_modules/jest-circus/build/run.js:212:5)
    at _runTest (/dev/shm/workspace/parallel/3/kibana/node_modules/jest-circus/build/run.js:149:3)
    at _runTestsForDescribeBlock (/dev/shm/workspace/parallel/3/kibana/node_modules/jest-circus/build/run.js:63:9)
    at _runTestsForDescribeBlock (/dev/shm/workspace/parallel/3/kibana/node_modules/jest-circus/build/run.js:57:9)
    at _runTestsForDescribeBlock (/dev/shm/workspace/parallel/3/kibana/node_modules/jest-circus/build/run.js:57:9)
    at run (/dev/shm/workspace/parallel/3/kibana/node_modules/jest-circus/build/run.js:25:3)
    at runAndTransformResultsToJestFormat (/dev/shm/workspace/parallel/3/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:176:21)
    at jestAdapter (/dev/shm/workspace/parallel/3/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:109:19)
    at runTestInternal (/dev/shm/workspace/parallel/3/kibana/node_modules/jest-runner/build/runTest.js:380:16)
    at runTest (/dev/shm/workspace/parallel/3/kibana/node_modules/jest-runner/build/runTest.js:472:34)

First failure: Jenkins Build

@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Jun 24, 2021
@botelastic botelastic bot added the needs-team Issues missing a team label label Jun 24, 2021
@jportner jportner added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc and removed needs-team Issues missing a team label labels Jun 29, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@rudolf
Copy link
Contributor

rudolf commented Sep 15, 2021

Closing as there seems to have been only a single failure

@rudolf rudolf closed this as completed Sep 15, 2021
@kibanamachine kibanamachine reopened this Sep 22, 2021
@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@jbudz
Copy link
Member

jbudz commented Sep 23, 2021

skipped

main: 0484e6c
7.x: 698d923

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@rudolf
Copy link
Contributor

rudolf commented Sep 29, 2021

To fix the initial failures I used a data archive with a lot more data which avoided the race condition where the reindex operation completes in "0s". But now tests seem to be flaky because the larger index takes longer to become available. So I'll be adding a check to only start the reindex once the larger index is "yellow".

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@mistic mistic closed this as completed Oct 4, 2021
@kibanamachine kibanamachine reopened this Oct 4, 2021
@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 7.16

@exalate-issue-sync exalate-issue-sync bot added impact:critical This issue should be addressed immediately due to a critical level of impact on the product. loe:small Small Level of Effort loe:medium Medium Level of Effort loe:large Large Level of Effort loe:x-large Extra Large Level of Effort and removed loe:small Small Level of Effort loe:medium Medium Level of Effort loe:large Large Level of Effort loe:x-large Extra Large Level of Effort labels Dec 14, 2021
@TinaHeiligers
Copy link
Contributor

TinaHeiligers commented Dec 15, 2021

This test seems to be passing on today's hourly builds for 7.16 and 8.0. Based on the history of the flakiness here, should we set time aside to re-evaluate our testing strategy for integration tests?

@rudolf
Copy link
Contributor

rudolf commented Feb 8, 2022

The last failure doesn't seem to be flakiness but someone trying to build 7.16 with an 7.17 ES version before we added the necessary support. This created the following errors:

Snapshots for 7.16.1 are available, but couldn't find an artifact in the manifest for [7.16.1, default, linux]

Closing, since the last fix all failures turned out to be noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Feature:Saved Objects impact:critical This issue should be addressed immediately due to a critical level of impact on the product. Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
7 participants