Failing test: License plugin API Integration Tests.x-pack/test/licensing_plugin/public/updates·ts - Licensing plugin public client changes in license types provides changes in license types

[kibanamachine]

A test failed on a tracked branch

{ JavascriptError: javascript error: Service Unavailable
JavaScript stack:
Error: Service Unavailable
    at _callee3$ (http://localhost:6131/bundles/commons.bundle.js:12770:23)
    at tryCatch (http://localhost:6131/built_assets/dlls/vendors.bundle.dll.js:504:570349)
    at Generator.invoke [as _invoke] (http://localhost:6131/built_assets/dlls/vendors.bundle.dll.js:504:574198)
    at Generator.forEach.prototype.<computed> [as next] (http://localhost:6131/built_assets/dlls/vendors.bundle.dll.js:504:571473)
    at asyncGeneratorStep (http://localhost:6131/bundles/commons.bundle.js:12468:24)
    at _next (http://localhost:6131/bundles/commons.bundle.js:12490:9)
  (Session info: headless chrome=79.0.3945.88)
  (Driver info: chromedriver=79.0.3945.36 (3582db32b33893869b8c1339e8f4d9ed1816f143-refs/branch-heads/3945@{#614}),platform=Linux 4.9.0-11-amd64 x86_64)
    at Object.checkLegacyResponse (/dev/shm/workspace/kibana/node_modules/selenium-webdriver/lib/error.js:585:15)
    at parseHttpResponse (/dev/shm/workspace/kibana/node_modules/selenium-webdriver/lib/http.js:554:13)
    at Executor.execute (/dev/shm/workspace/kibana/node_modules/selenium-webdriver/lib/http.js:489:26)
    at process._tickCallback (internal/process/next_tick.js:68:7) name: 'JavascriptError', remoteStacktrace: '' }

First failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-test-triage (failed-test)

[mshustov]

#53002 reverted

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[tsullivan]

I am seeing flakiness with this test, and it looks like a different error message result:

https://kibana-ci.elastic.co/job/elastic+kibana+pipeline-pull-request/120699/testReport/License%20plugin%20API%20Integration%20Tests/x-pack_test_licensing_plugin_public_updates%C2%B7ts/Kibana_Pipeline___general___Licensing_plugin_public_client_changes_in_license_types_provides_changes_in_license_types/

[00:00:31]               │ info Saving page source to: /dev/shm/workspace/parallel/3/kibana/x-pack/test/licensing_plugin/failure_debug/html/Licensing plugin public client changes in license types provides changes in license types.html
[00:00:31]               └- ✖ fail: Licensing plugin public client changes in license types provides changes in license types
[00:00:31]               │      Error: expected 'basic' to equal 'trial'
[00:00:31]               │       at Assertion.assert (/dev/shm/workspace/parallel/3/kibana/node_modules/@kbn/expect/expect.js:100:11)
[00:00:31]               │       at Assertion.be.Assertion.equal (/dev/shm/workspace/parallel/3/kibana/node_modules/@kbn/expect/expect.js:227:8)
[00:00:31]               │       at Assertion.be (/dev/shm/workspace/parallel/3/kibana/node_modules/@kbn/expect/expect.js:69:22)
[00:00:31]               │       at Context.<anonymous> (test/licensing_plugin/public/updates.ts:74:12)
[00:00:31]               │       at runMicrotasks (<anonymous>)
[00:00:31]               │       at processTicksAndRejections (internal/process/task_queues.js:93:5)
[00:00:31]               │       at Object.apply (/dev/shm/workspace/parallel/3/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)
[00:00:31]               │ 
[00:00:31]               │ 

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[kibanamachine]

New failure: Jenkins Build

[tsullivan]

It's definitely happening because of #94966 which has not yet been backported to 7.x.

I tried to add a change that would stop this test from being flaky, but it looks like it wasn't enough (sorry about that @mistic): 2e732f2

In the backport branch, I am going to add the skip for this test as well, so it will be skipped in 7.x

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[rudolf]

It's definitely happening because of #94966

@tsullivan Could you please prioritise fixing this?

[elasticmachine]

Pinging @elastic/appex-sharedux (Team:SharedUX)

[tsullivan]

It's definitely happening because of #94966

@tsullivan Could you please prioritise fixing this?

@rudolf I'm getting stuck on this:
https://github.com/elastic/kibana/blob/3730dd0/x-pack/test/licensing_plugin/scenario.ts#L62-L69

From the code:

// elasticsearch allows to request trial only once. other attempts will throw 403.

In local development, I can get the test to pass the first time, but have to restart the server to get it to pass again. Without a restart, the test fails because the request that was expected to return a 200 returns a 403.

Question: How is this expected to work in a flaky test runner?

Also, I don't recall the original reason why I thought these test failures were related to my PR, but I vaguely remember not being able to get this test passing when working on that PR.

[tsullivan]

In local development, I can get the test to pass the first time, but have to restart the server to get it to pass again. Without a restart, the test fails because the request that was expected to return a 200 returns a 403.

Question: How is this expected to work in a flaky test runner?

Scratch that, the only failures I ended up seeing were due to an expired license coded into a test.

I have a fix ready for unskipping this here: #163199