Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/spaces/spaces_selection·ts - Spaces app Spaces Spaces Data "before all" hook in "Spaces Data" #52714

Closed
kibanamachine opened this issue Dec 11, 2019 · 16 comments · Fixed by #147519 or #198921
Closed

Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/spaces/spaces_selection·ts - Spaces app Spaces Spaces Data "before all" hook in "Spaces Data" #52714

kibanamachine opened this issue Dec 11, 2019 · 16 comments · Fixed by #147519 or #198921
Assignees
@jeramysoucy
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Security Team focused on: Auth, Users, Roles, Spaces, Audit Logging, and more!

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Dec 11, 2019 •
edited
Loading

A test failed on a tracked branch

Error: retry.tryForTime timeout: Error: retry.try timeout: Error: App failed to load: login in 10000ms appUrl=http://localhost:6151/login currentUrl=http://localhost:6151/spaces/space_selector
    at retry.try (/dev/shm/workspace/kibana/test/functional/page_objects/common_page.ts:250:19)
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at onFailure (/dev/shm/workspace/kibana/test/common/services/retry/retry_for_success.ts:28:9)
    at retryForSuccess (/dev/shm/workspace/kibana/test/common/services/retry/retry_for_success.ts:68:13)
    at onFailure (/dev/shm/workspace/kibana/test/common/services/retry/retry_for_success.ts:28:9)
    at retryForSuccess (/dev/shm/workspace/kibana/test/common/services/retry/retry_for_success.ts:68:13)

First failure: Jenkins Build
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Dec 11, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-test-triage (failed-test)

@azasypkin azasypkin added the Team:Security Team focused on: Auth, Users, Roles, Spaces, Audit Logging, and more! label Dec 11, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-security (Team:Security)

@kobelb
Copy link
Contributor

kobelb commented Feb 27, 2020

This test failed once almost 3 months ago, closing

@kobelb kobelb closed this as completed Feb 27, 2020
@kibanamachine kibanamachine reopened this Nov 16, 2022
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.6

@jeramysoucy jeramysoucy mentioned this issue Dec 14, 2022
Merged
jeramysoucy added a commit that referenced this issue Dec 15, 2022
@jeramysoucy @kibanamachine
[Failing Test] Fixes Spaces Data Before All Failure (#147519)
c2be109 
Fixes #52714
Fixes #52715

After all failure likely due to before failure - when before fails, the
data set is not installed and cannot be removed.

Before all failure seems likely to be a race condition of one or more
elements being 'not interactable'. I was unable to reproduce the issue
locally and via a [Flaky Test
Runner](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/1646),
but walking through the code noticed an incorrect attribute reference
and a few places where we could improve determinism.
- Replaced 'showSampleDataAccordion'->'class' check with
'showSampleDataButton'->'aria-expanded'
- Added deterministic verification of sample data accordion open
- Mirrored previous timing/enable check fix to 'removeSampleDataSet' in
'addSampleDataSet'
- Improved determinism in 'removeSampleDataSet', adding an initial check
for whether the data set was installed
- Added additional debug logs to improve any future troubleshooting

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@kibanamachine kibanamachine reopened this Mar 27, 2024
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@azasypkin
Copy link
Member

Let's run flaky test runner and close it if FTR is green (it seems to be very infrequent).

@elena-shostak elena-shostak self-assigned this Aug 8, 2024
@elena-shostak
Copy link
Contributor

FTR run is green, closing this one.

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@kibanamachine kibanamachine reopened this Sep 20, 2024
@legrego
Copy link
Member

legrego commented Oct 1, 2024

Transient failure, closing.

@legrego legrego closed this as completed Oct 1, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@kibanamachine kibanamachine reopened this Oct 4, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.16

@jeramysoucy
Copy link
Contributor

Latest failures appear to be due to sample data already being installed when the test is looking for the "add data" button.
Image

@elena-shostak elena-shostak removed their assignment Oct 28, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@jeramysoucy jeramysoucy self-assigned this Oct 31, 2024
@jeramysoucy jeramysoucy mentioned this issue Nov 5, 2024
Merged
jeramysoucy added a commit that referenced this issue Nov 5, 2024
@jeramysoucy
Improves cleanup in spaces data functional test (#198921)
b5c8ed7 
Closes #52714

## Summary

Uses the sample data FTR service to remove the sample data via API
during cleanup. Previously, the after block only removed the sample data
from one space and attempted to rely on an esArchiver call to clean up
remaining data later on. As the scope of the test does not use the
esArchiver any longer, it seems more appropriate to remove the sample
data more deterministically.

### Flaky test runner

-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 5, 2024
@jeramysoucy
Improves cleanup in spaces data functional test (elastic#198921)
fd07c95 
Closes elastic#52714

## Summary

Uses the sample data FTR service to remove the sample data via API
during cleanup. Previously, the after block only removed the sample data
from one space and attempted to rely on an esArchiver call to clean up
remaining data later on. As the scope of the test does not use the
esArchiver any longer, it seems more appropriate to remove the sample
data more deterministically.

### Flaky test runner

-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335

(cherry picked from commit b5c8ed7)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 5, 2024
@jeramysoucy
Improves cleanup in spaces data functional test (elastic#198921)
66742cb 
Closes elastic#52714

## Summary

Uses the sample data FTR service to remove the sample data via API
during cleanup. Previously, the after block only removed the sample data
from one space and attempted to rely on an esArchiver call to clean up
remaining data later on. As the scope of the test does not use the
esArchiver any longer, it seems more appropriate to remove the sample
data more deterministically.

### Flaky test runner

-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335

(cherry picked from commit b5c8ed7)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 5, 2024
@jeramysoucy
Improves cleanup in spaces data functional test (elastic#198921)
bebeb2e 
Closes elastic#52714

## Summary

Uses the sample data FTR service to remove the sample data via API
during cleanup. Previously, the after block only removed the sample data
from one space and attempted to rely on an esArchiver call to clean up
remaining data later on. As the scope of the test does not use the
esArchiver any longer, it seems more appropriate to remove the sample
data more deterministically.

### Flaky test runner

-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335

(cherry picked from commit b5c8ed7)
kibanamachine added a commit that referenced this issue Nov 5, 2024
@kibanamachine @jeramysoucy
[8.x] Improves cleanup in spaces data functional test (#198921) (#199024
6fa817a 
)

# Backport

This will backport the following commits from `main` to `8.x`:
- [Improves cleanup in spaces data functional test
(#198921)](#198921)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jeramy
Soucy","email":"jeramy.soucy@elastic.co"},"sourceCommit":{"committedDate":"2024-11-05T16:31:25Z","message":"Improves
cleanup in spaces data functional test (#198921)\n\nCloses
#52714\r\n\r\n## Summary\r\n\r\nUses the sample data FTR service to
remove the sample data via API\r\nduring cleanup. Previously, the after
block only removed the sample data\r\nfrom one space and attempted to
rely on an esArchiver call to clean up\r\nremaining data later on. As
the scope of the test does not use the\r\nesArchiver any longer, it
seems more appropriate to remove the sample\r\ndata more
deterministically.\r\n\r\n### Flaky test
runner\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335","sha":"b5c8ed7b5c8883993a534626384fe92b728e850c","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Security","release_note:skip","v9.0.0","backport:all-open","FTR"],"title":"Improves
cleanup in spaces data functional
test","number":198921,"url":"https://github.com/elastic/kibana/pull/198921","mergeCommit":{"message":"Improves
cleanup in spaces data functional test (#198921)\n\nCloses
#52714\r\n\r\n## Summary\r\n\r\nUses the sample data FTR service to
remove the sample data via API\r\nduring cleanup. Previously, the after
block only removed the sample data\r\nfrom one space and attempted to
rely on an esArchiver call to clean up\r\nremaining data later on. As
the scope of the test does not use the\r\nesArchiver any longer, it
seems more appropriate to remove the sample\r\ndata more
deterministically.\r\n\r\n### Flaky test
runner\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335","sha":"b5c8ed7b5c8883993a534626384fe92b728e850c"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/198921","number":198921,"mergeCommit":{"message":"Improves
cleanup in spaces data functional test (#198921)\n\nCloses
#52714\r\n\r\n## Summary\r\n\r\nUses the sample data FTR service to
remove the sample data via API\r\nduring cleanup. Previously, the after
block only removed the sample data\r\nfrom one space and attempted to
rely on an esArchiver call to clean up\r\nremaining data later on. As
the scope of the test does not use the\r\nesArchiver any longer, it
seems more appropriate to remove the sample\r\ndata more
deterministically.\r\n\r\n### Flaky test
runner\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335","sha":"b5c8ed7b5c8883993a534626384fe92b728e850c"}}]}]
BACKPORT-->

Co-authored-by: Jeramy Soucy <jeramy.soucy@elastic.co>
kibanamachine added a commit that referenced this issue Nov 5, 2024
@kibanamachine @jeramysoucy
[8.15] Improves cleanup in spaces data functional test (#198921) (#19…
352f992 
…9022)

# Backport

This will backport the following commits from `main` to `8.15`:
- [Improves cleanup in spaces data functional test
(#198921)](#198921)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jeramy
Soucy","email":"jeramy.soucy@elastic.co"},"sourceCommit":{"committedDate":"2024-11-05T16:31:25Z","message":"Improves
cleanup in spaces data functional test (#198921)\n\nCloses
#52714\r\n\r\n## Summary\r\n\r\nUses the sample data FTR service to
remove the sample data via API\r\nduring cleanup. Previously, the after
block only removed the sample data\r\nfrom one space and attempted to
rely on an esArchiver call to clean up\r\nremaining data later on. As
the scope of the test does not use the\r\nesArchiver any longer, it
seems more appropriate to remove the sample\r\ndata more
deterministically.\r\n\r\n### Flaky test
runner\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335","sha":"b5c8ed7b5c8883993a534626384fe92b728e850c","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Security","release_note:skip","v9.0.0","backport:all-open","FTR"],"title":"Improves
cleanup in spaces data functional
test","number":198921,"url":"https://github.com/elastic/kibana/pull/198921","mergeCommit":{"message":"Improves
cleanup in spaces data functional test (#198921)\n\nCloses
#52714\r\n\r\n## Summary\r\n\r\nUses the sample data FTR service to
remove the sample data via API\r\nduring cleanup. Previously, the after
block only removed the sample data\r\nfrom one space and attempted to
rely on an esArchiver call to clean up\r\nremaining data later on. As
the scope of the test does not use the\r\nesArchiver any longer, it
seems more appropriate to remove the sample\r\ndata more
deterministically.\r\n\r\n### Flaky test
runner\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335","sha":"b5c8ed7b5c8883993a534626384fe92b728e850c"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/198921","number":198921,"mergeCommit":{"message":"Improves
cleanup in spaces data functional test (#198921)\n\nCloses
#52714\r\n\r\n## Summary\r\n\r\nUses the sample data FTR service to
remove the sample data via API\r\nduring cleanup. Previously, the after
block only removed the sample data\r\nfrom one space and attempted to
rely on an esArchiver call to clean up\r\nremaining data later on. As
the scope of the test does not use the\r\nesArchiver any longer, it
seems more appropriate to remove the sample\r\ndata more
deterministically.\r\n\r\n### Flaky test
runner\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335","sha":"b5c8ed7b5c8883993a534626384fe92b728e850c"}}]}]
BACKPORT-->

Co-authored-by: Jeramy Soucy <jeramy.soucy@elastic.co>
kibanamachine added a commit that referenced this issue Nov 5, 2024
@kibanamachine @jeramysoucy
[8.16] Improves cleanup in spaces data functional test (#198921) (#19…
652054e 
…9023)

# Backport

This will backport the following commits from `main` to `8.16`:
- [Improves cleanup in spaces data functional test
(#198921)](#198921)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jeramy
Soucy","email":"jeramy.soucy@elastic.co"},"sourceCommit":{"committedDate":"2024-11-05T16:31:25Z","message":"Improves
cleanup in spaces data functional test (#198921)\n\nCloses
#52714\r\n\r\n## Summary\r\n\r\nUses the sample data FTR service to
remove the sample data via API\r\nduring cleanup. Previously, the after
block only removed the sample data\r\nfrom one space and attempted to
rely on an esArchiver call to clean up\r\nremaining data later on. As
the scope of the test does not use the\r\nesArchiver any longer, it
seems more appropriate to remove the sample\r\ndata more
deterministically.\r\n\r\n### Flaky test
runner\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335","sha":"b5c8ed7b5c8883993a534626384fe92b728e850c","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Security","release_note:skip","v9.0.0","backport:all-open","FTR"],"title":"Improves
cleanup in spaces data functional
test","number":198921,"url":"https://github.com/elastic/kibana/pull/198921","mergeCommit":{"message":"Improves
cleanup in spaces data functional test (#198921)\n\nCloses
#52714\r\n\r\n## Summary\r\n\r\nUses the sample data FTR service to
remove the sample data via API\r\nduring cleanup. Previously, the after
block only removed the sample data\r\nfrom one space and attempted to
rely on an esArchiver call to clean up\r\nremaining data later on. As
the scope of the test does not use the\r\nesArchiver any longer, it
seems more appropriate to remove the sample\r\ndata more
deterministically.\r\n\r\n### Flaky test
runner\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335","sha":"b5c8ed7b5c8883993a534626384fe92b728e850c"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/198921","number":198921,"mergeCommit":{"message":"Improves
cleanup in spaces data functional test (#198921)\n\nCloses
#52714\r\n\r\n## Summary\r\n\r\nUses the sample data FTR service to
remove the sample data via API\r\nduring cleanup. Previously, the after
block only removed the sample data\r\nfrom one space and attempted to
rely on an esArchiver call to clean up\r\nremaining data later on. As
the scope of the test does not use the\r\nesArchiver any longer, it
seems more appropriate to remove the sample\r\ndata more
deterministically.\r\n\r\n### Flaky test
runner\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335","sha":"b5c8ed7b5c8883993a534626384fe92b728e850c"}}]}]
BACKPORT-->

Co-authored-by: Jeramy Soucy <jeramy.soucy@elastic.co>
mgadewoll pushed a commit to mgadewoll/kibana that referenced this issue Nov 7, 2024
@jeramysoucy @mgadewoll
Improves cleanup in spaces data functional test (elastic#198921)
2ec0266 
Closes elastic#52714

## Summary

Uses the sample data FTR service to remove the sample data via API
during cleanup. Previously, the after block only removed the sample data
from one space and attempted to rely on an esArchiver call to clean up
remaining data later on. As the scope of the test does not use the
esArchiver any longer, it seems more appropriate to remove the sample
data more deterministically.

### Flaky test runner

-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7335
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jeramysoucy jeramysoucy

Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Security Team focused on: Auth, Users, Roles, Spaces, Audit Logging, and more!
Projects
None yet
Milestone
No milestone
7 participants
@kobelb @azasypkin @legrego @elasticmachine @kibanamachine @jeramysoucy @elena-shostak