Skip to content

Commit

Permalink
[8.15] fix for flaky sample data test (#199005) (#199049)
Browse files Browse the repository at this point in the history
# Backport

This will backport the following commits from `main` to `8.15`:
- [fix for flaky sample data test
(#199005)](#199005)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Eyo O.
Eyo","email":"7893459+eokoneyo@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-11-05T20:00:03Z","message":"fix
for flaky sample data test (#199005)\n\n## Summary\r\n\r\nCloses
https://github.com/elastic/kibana/issues/187473,\r\nhttps://github.com/elastic/kibana/issues/112103\r\n\r\n###
Checklist\r\n<!--\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [ ] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [ ] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios -->\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n<!--\r\n- [ ] Any UI touched in
this PR is usable by keyboard only (learn more\r\nabout [keyboard
accessibility](https://webaim.org/techniques/keyboard/))\r\n- [ ] Any UI
touched in this PR does not create any new axe failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[ ] If a plugin configuration key changed, check if it needs to
be\r\nallowlisted in the cloud and added to the
[docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n-
[ ] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[ ] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n###
Risk Matrix\r\n\r\nDelete this section if it is not applicable to this
PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other
developers to\r\nidentify risks that should be tested prior to the
change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider
some of the following examples\r\nand how they may potentially impact
the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes
|\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n|
Multiple Spaces&mdash;unexpected behavior in non-default Kibana
Space.\r\n| Low | High | Integration tests will verify that all features
are still\r\nsupported in non-default Kibana Space and when user
switches between\r\nspaces. |\r\n| Multiple nodes&mdash;Elasticsearch
polling might have race conditions\r\nwhen multiple Kibana nodes are
polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so
executing them multiple times will not result\r\nin logical error, but
will degrade performance. To test for this case we\r\nadd plenty of unit
tests around this logic and document manual testing\r\nprocedure. |\r\n|
Code should gracefully handle cases when feature X or plugin Y
are\r\ndisabled. | Medium | High | Unit tests will verify that any
feature flag\r\nor plugin combination still results in our service
operational. |\r\n| [See more potential
risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
|\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for
breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n-
[ ] This will appear in the **Release Notes** and follow
the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n-->","sha":"f988de272608bcb93b8a22f35c945f28dff50a91","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:SharedUX","backport:prev-major"],"title":"fix
for flaky sample data
test","number":199005,"url":"https://github.com/elastic/kibana/pull/199005","mergeCommit":{"message":"fix
for flaky sample data test (#199005)\n\n## Summary\r\n\r\nCloses
https://github.com/elastic/kibana/issues/187473,\r\nhttps://github.com/elastic/kibana/issues/112103\r\n\r\n###
Checklist\r\n<!--\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [ ] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [ ] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios -->\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n<!--\r\n- [ ] Any UI touched in
this PR is usable by keyboard only (learn more\r\nabout [keyboard
accessibility](https://webaim.org/techniques/keyboard/))\r\n- [ ] Any UI
touched in this PR does not create any new axe failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[ ] If a plugin configuration key changed, check if it needs to
be\r\nallowlisted in the cloud and added to the
[docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n-
[ ] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[ ] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n###
Risk Matrix\r\n\r\nDelete this section if it is not applicable to this
PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other
developers to\r\nidentify risks that should be tested prior to the
change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider
some of the following examples\r\nand how they may potentially impact
the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes
|\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n|
Multiple Spaces&mdash;unexpected behavior in non-default Kibana
Space.\r\n| Low | High | Integration tests will verify that all features
are still\r\nsupported in non-default Kibana Space and when user
switches between\r\nspaces. |\r\n| Multiple nodes&mdash;Elasticsearch
polling might have race conditions\r\nwhen multiple Kibana nodes are
polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so
executing them multiple times will not result\r\nin logical error, but
will degrade performance. To test for this case we\r\nadd plenty of unit
tests around this logic and document manual testing\r\nprocedure. |\r\n|
Code should gracefully handle cases when feature X or plugin Y
are\r\ndisabled. | Medium | High | Unit tests will verify that any
feature flag\r\nor plugin combination still results in our service
operational. |\r\n| [See more potential
risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
|\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for
breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n-
[ ] This will appear in the **Release Notes** and follow
the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n-->","sha":"f988de272608bcb93b8a22f35c945f28dff50a91"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199005","number":199005,"mergeCommit":{"message":"fix
for flaky sample data test (#199005)\n\n## Summary\r\n\r\nCloses
https://github.com/elastic/kibana/issues/187473,\r\nhttps://github.com/elastic/kibana/issues/112103\r\n\r\n###
Checklist\r\n<!--\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [ ] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [ ] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios -->\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n<!--\r\n- [ ] Any UI touched in
this PR is usable by keyboard only (learn more\r\nabout [keyboard
accessibility](https://webaim.org/techniques/keyboard/))\r\n- [ ] Any UI
touched in this PR does not create any new axe failures\r\n(run axe in
browser:\r\n[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),\r\n[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))\r\n-
[ ] If a plugin configuration key changed, check if it needs to
be\r\nallowlisted in the cloud and added to the
[docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n-
[ ] This renders correctly on smaller devices using a
responsive\r\nlayout. (You can test this [in
your\r\nbrowser](https://www.browserstack.com/guide/responsive-testing-on-local-server))\r\n-
[ ] This was checked for
[cross-browser\r\ncompatibility](https://www.elastic.co/support/matrix#matrix_browsers)\r\n\r\n\r\n###
Risk Matrix\r\n\r\nDelete this section if it is not applicable to this
PR.\r\n\r\nBefore closing this PR, invite QA, stakeholders, and other
developers to\r\nidentify risks that should be tested prior to the
change/feature\r\nrelease.\r\n\r\nWhen forming the risk matrix, consider
some of the following examples\r\nand how they may potentially impact
the change:\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes
|\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n|
Multiple Spaces&mdash;unexpected behavior in non-default Kibana
Space.\r\n| Low | High | Integration tests will verify that all features
are still\r\nsupported in non-default Kibana Space and when user
switches between\r\nspaces. |\r\n| Multiple nodes&mdash;Elasticsearch
polling might have race conditions\r\nwhen multiple Kibana nodes are
polling for the same tasks. | High | Low\r\n| Tasks are idempotent, so
executing them multiple times will not result\r\nin logical error, but
will degrade performance. To test for this case we\r\nadd plenty of unit
tests around this logic and document manual testing\r\nprocedure. |\r\n|
Code should gracefully handle cases when feature X or plugin Y
are\r\ndisabled. | Medium | High | Unit tests will verify that any
feature flag\r\nor plugin combination still results in our service
operational. |\r\n| [See more potential
risk\r\nexamples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
|\r\n\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for
breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n-
[ ] This will appear in the **Release Notes** and follow
the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n-->","sha":"f988de272608bcb93b8a22f35c945f28dff50a91"}}]}]
BACKPORT-->

Co-authored-by: Eyo O. Eyo <7893459+eokoneyo@users.noreply.github.com>
  • Loading branch information
kibanamachine and eokoneyo authored Nov 5, 2024
1 parent 352f992 commit 6fa254e
Showing 1 changed file with 34 additions and 28 deletions.
62 changes: 34 additions & 28 deletions test/functional/page_objects/home_page.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,40 +85,46 @@ export class HomePageObject extends FtrService {

async addSampleDataSet(id: string) {
await this.openSampleDataAccordion();
const isInstalled = await this.isSampleDataSetInstalled(id);
if (!isInstalled) {
await this.retry.waitFor('sample data to be installed', async () => {
// count for the edge case where some how installation completes just before the retry occurs
if (await this.isSampleDataSetInstalled(id)) {
return true;
}

this.log.debug(`Attempting to add sample data: ${id}`);
await this.retry.waitFor('sample data to be installed', async () => {
// Echoing the adjustments made to 'removeSampleDataSet', as we are seeing flaky test cases here as well
// https://github.com/elastic/kibana/issues/52714
await this.testSubjects.waitForEnabled(`addSampleDataSet${id}`);
await this.common.sleep(1010);
await this.testSubjects.click(`addSampleDataSet${id}`);
await this.common.sleep(1010);
await this._waitForSampleDataLoadingAction(id);
return await this.isSampleDataSetInstalled(id);
});
}

// Echoing the adjustments made to 'removeSampleDataSet', as we are seeing flaky test cases here as well
// https://github.com/elastic/kibana/issues/52714
await this.testSubjects.waitForEnabled(`addSampleDataSet${id}`);
await this.common.sleep(1010);
await this.testSubjects.click(`addSampleDataSet${id}`);
await this.common.sleep(1010);
await this._waitForSampleDataLoadingAction(id);
return await this.isSampleDataSetInstalled(id);
});
}

async removeSampleDataSet(id: string) {
await this.openSampleDataAccordion();
const isInstalled = await this.isSampleDataSetInstalled(id);
if (isInstalled) {
await this.retry.waitFor('sample data to be removed', async () => {
// account for the edge case where some how data is uninstalled just before the retry occurs
if (!(await this.isSampleDataSetInstalled(id))) {
return true;
}

this.log.debug(`Attempting to remove sample data: ${id}`);
await this.retry.waitFor('sample data to be removed', async () => {
// looks like overkill but we're hitting flaky cases where we click but it doesn't remove
await this.testSubjects.waitForEnabled(`removeSampleDataSet${id}`);
// https://github.com/elastic/kibana/issues/65949
// Even after waiting for the "Remove" button to be enabled we still have failures
// where it appears the click just didn't work.
await this.common.sleep(1010);
await this.testSubjects.click(`removeSampleDataSet${id}`);
await this.common.sleep(1010);
await this._waitForSampleDataLoadingAction(id);
return !(await this.isSampleDataSetInstalled(id));
});
}

// looks like overkill but we're hitting flaky cases where we click but it doesn't remove
await this.testSubjects.waitForEnabled(`removeSampleDataSet${id}`);
// https://github.com/elastic/kibana/issues/65949
// Even after waiting for the "Remove" button to be enabled we still have failures
// where it appears the click just didn't work.
await this.common.sleep(1010);
await this.testSubjects.click(`removeSampleDataSet${id}`);
await this.common.sleep(1010);
await this._waitForSampleDataLoadingAction(id);
return !(await this.isSampleDataSetInstalled(id));
});
}

// loading action is either uninstall and install
Expand Down

0 comments on commit 6fa254e

Please sign in to comment.