Failing test: Chrome UI Functional Tests.test/functional/apps/saved_objects_management/show_relationships·ts - saved objects management saved objects relationships flyout displays the invalid references

[kibanamachine]

A test failed on a tracked branch

Error: expected [ { type: 'dashboard',
    id: 'missing-dashboard-ref',
    relationship: 'Child',
    error: 'Saved object [dashboard/missing-dashboard-ref] not found' } ] to sort of equal [ { error: 'Saved object [visualization/missing-vis-ref] not found',
    id: 'missing-vis-ref',
    relationship: 'Child',
    type: 'visualization' },
  { error: 'Saved object [dashboard/missing-dashboard-ref] not found',
    id: 'missing-dashboard-ref',
    relationship: 'Child',
    type: 'dashboard' } ]
    at Assertion.assert (node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.eql (node_modules/@kbn/expect/expect.js:244:8)
    at Context.<anonymous> (test/functional/apps/saved_objects_management/show_relationships.ts:40:35)
    at Object.apply (node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:78:16) {
  actual: '[\n' +
    '  {\n' +
    '    "error": "Saved object [dashboard/missing-dashboard-ref] not found"\n' +
    '    "id": "missing-dashboard-ref"\n' +
    '    "relationship": "Child"\n' +
    '    "type": "dashboard"\n' +
    '  }\n' +
    ']',
  expected: '[\n' +
    '  {\n' +
    '    "error": "Saved object [visualization/missing-vis-ref] not found"\n' +
    '    "id": "missing-vis-ref"\n' +
    '    "relationship": "Child"\n' +
    '    "type": "visualization"\n' +
    '  }\n' +
    '  {\n' +
    '    "error": "Saved object [dashboard/missing-dashboard-ref] not found"\n' +
    '    "id": "missing-dashboard-ref"\n' +
    '    "relationship": "Child"\n' +
    '    "type": "dashboard"\n' +
    '  }\n' +
    ']',
  showDiff: true
}

First failure: CI Build - 7.17

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[kibanamachine]

New failure: CI Build - 8.14

[afharo]

The recent error looks like a webdriver hiccup:

│ WebDriverError: disconnected: not connected to DevTools
--
  | │   (failed to check if window was closed: disconnected: not connected to DevTools)
  | │   (Session info: chrome=124.0.6367.118)

Closing for now.

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - 7.17

[kibanamachine]

New failure: kibana-on-merge - 7.17

[jloleysens]

Looks like some kind of unstable array sorting causing an issue in displays the invalid references

│1)    saved objects management
--
  | │       saved objects relationships flyout
  | │         displays the invalid references:
  | │
  | │
  | │       Error: expected [ { type: 'dashboard',
  | │     id: 'missing-dashboard-ref',
  | │     relationship: 'Child',
  | │     error: 'Saved object [dashboard/missing-dashboard-ref] not found' } ] to sort of equal [ { error: 'Saved object [visualization/missing-vis-ref] not found',
  | │     id: 'missing-vis-ref',
  | │     relationship: 'Child',
  | │     type: 'visualization' },
  | │   { error: 'Saved object [dashboard/missing-dashboard-ref] not found',
  | │     id: 'missing-dashboard-ref',
  | │     relationship: 'Child',
  | │     type: 'dashboard' } ]
  | │       + expected - actual
  | │
  | │        [
  | │          {
  | │       +    "error": "Saved object [visualization/missing-vis-ref] not found"
  | │       +    "id": "missing-vis-ref"
  | │       +    "relationship": "Child"
  | │       +    "type": "visualization"
  | │       +  }
  | │       +  {
  | │            "error": "Saved object [dashboard/missing-dashboard-ref] not found"
  | │            "id": "missing-dashboard-ref"
  | │            "relationship": "Child"
  | │            "type": "dashboard"

[kibanamachine]

New failure: kibana-on-merge - 7.17

[kibanamachine]

New failure: kibana-on-merge - 7.17

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - 7.17

[kibanamachine]

New failure: kibana-on-merge - 7.17

[kibanamachine]

New failure: kibana-on-merge - 7.17

[kibanamachine]

New failure: kibana-on-merge - 7.17

[kibanamachine]

Skipped

7.17: faae6da

[gsoldevila]

If we look at the artifacts produced on the failed CI test, we can see that the 2 SOs with invalid references are on the list:

So IMO this could be a problem with the await PageObjects.savedObjects.getInvalidRelations(); implementation.

Will investigate a bit more.

[gsoldevila]

@jloleysens Sorry I just realised the failure + skip was before your commit.
Nonetheless, main does not have this flaky behaviour and they are not using a retry mechanism.

What I noticed is that Bluebird's mapAsync as replaced by native Promise.all with this PR.

That's the only noticeable difference between 7.x and 8.x, so I'd say it's worth updating the test with this change instead. WDYT?

[jloleysens]

Hey @gsoldevila I was not aware of this PR

I'm not sure bluebird is the reason we needed the retry logic... There might be something else and so would be good to put any future changes through flaky test runner. In principle happy to go ahead with your approach though!

Also, not sure why my PR didn't close this issue 😅 .

[gsoldevila]

Triggered a 300x FTR
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6649

UPDATE: getting rid of bluebird does not fix the issue.

I still don't know why it does NOT fail on 8.x 🤔

[gsoldevila]

JLs fix should have closed this issue automatically.