[7.x] [Ingest] Support yaml variables in datasource (#64459)

[nchaulet]

Backports the following commits to 7.x:

• [Ingest] Support yaml variables in datasource ([Ingest] Support yaml variables in datasource #64459)

[kibanamachine]

💛 Build succeeded, but was flaky

• continuous-integration/kibana-ci/pull-request
• Commit: 88f4b52
• Flaky suites:
  • xpack-kibana-ciGroup2

---

Test Failures

Kibana Pipeline / kibana-xpack-agent / Plugin Functional Tests.x-pack/test/plugin_api_integration/test_suites/event_log/public_api_integration·ts.event_log Event Log public API should support pagination for events

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: event_log
[00:00:00]           └-> "before all" hook
[00:00:00]           └-: Event Log public API
[00:00:00]             └-> "before all" hook
[00:00:00]             └-> should allow querying for events by Saved Object
[00:00:00]               └-> "before each" hook: global before each
[00:00:00]               │ debg Logging Event for Saved Object 5f9d2397-4724-46ef-8bef-7b348fbbf257
[00:00:00]               │ proc [kibana]   log   [03:02:59.769] [info][eventLogFixture][event_log_fixture][plugins][plugins] test fixture: log event: 5f9d2397-4724-46ef-8bef-7b348fbbf257 {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"5f9d2397-4724-46ef-8bef-7b348fbbf257"}]},"message":"test 2020-04-29T03:02:59.680Z"}
[00:00:00]               │ proc [kibana]   log   [03:02:59.805] [info][eventLogFixture][event_log_fixture][plugins][plugins] log event error: Error: Saved object [event_log_test/5f9d2397-4724-46ef-8bef-7b348fbbf257] not found
[00:00:00]               │ proc [kibana]   log   [03:02:59.977] [info][eventLogFixture][event_log_fixture][plugins][plugins] created saved object
[00:00:00]               │ proc [kibana]   log   [03:02:59.978] [info][eventLogFixture][event_log_fixture][plugins][plugins] logged
[00:00:00]               │ debg Logging Event for Saved Object 5f9d2397-4724-46ef-8bef-7b348fbbf257
[00:00:00]               │ proc [kibana]   log   [03:02:59.980] [info][eventLog][plugins] event logged: {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"5f9d2397-4724-46ef-8bef-7b348fbbf257"}],"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"test 2020-04-29T03:02:59.680Z","@timestamp":"2020-04-29T03:02:59.977Z","ecs":{"version":"1.5.0"}}
[00:00:00]               │ proc [kibana]   log   [03:03:00.068] [info][eventLogFixture][event_log_fixture][plugins][plugins] test fixture: log event: 5f9d2397-4724-46ef-8bef-7b348fbbf257 {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"5f9d2397-4724-46ef-8bef-7b348fbbf257"}]},"message":"test 2020-04-29T03:02:59.680Z"}
[00:00:00]               │ proc [kibana]   log   [03:03:00.109] [info][eventLogFixture][event_log_fixture][plugins][plugins] found existing saved object
[00:00:00]               │ proc [kibana]   log   [03:03:00.110] [info][eventLogFixture][event_log_fixture][plugins][plugins] logged
[00:00:00]               │ debg calling /api/event_log/event_log_test/5f9d2397-4724-46ef-8bef-7b348fbbf257/_find
[00:00:00]               │ proc [kibana]   log   [03:03:00.112] [info][eventLog][plugins] event logged: {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"5f9d2397-4724-46ef-8bef-7b348fbbf257"}],"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"test 2020-04-29T03:02:59.680Z","@timestamp":"2020-04-29T03:03:00.109Z","ecs":{"version":"1.5.0"}}
[00:00:00]               │ debg --- retry.try error: expected 0 to equal 2
[00:00:01]               │ debg calling /api/event_log/event_log_test/5f9d2397-4724-46ef-8bef-7b348fbbf257/_find
[00:00:01]               └- ✓ pass  (1.1s) "event_log Event Log public API should allow querying for events by Saved Object"
[00:00:01]             └-> should support pagination for events
[00:00:01]               └-> "before each" hook: global before each
[00:00:01]               │ debg Logging Event for Saved Object 63cd19ee-ff34-4174-a07d-7ecd3667fd75
[00:00:01]               │ proc [kibana]   log   [03:03:00.892] [info][eventLogFixture][event_log_fixture][plugins][plugins] test fixture: log event: 63cd19ee-ff34-4174-a07d-7ecd3667fd75 {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}]},"message":"test 2020-04-29T03:03:00.829Z"}
[00:00:01]               │ proc [kibana]   log   [03:03:00.919] [info][eventLogFixture][event_log_fixture][plugins][plugins] log event error: Error: Saved object [event_log_test/63cd19ee-ff34-4174-a07d-7ecd3667fd75] not found
[00:00:01]               │ proc [kibana]   log   [03:03:00.990] [info][eventLogFixture][event_log_fixture][plugins][plugins] created saved object
[00:00:01]               │ proc [kibana]   log   [03:03:00.991] [info][eventLogFixture][event_log_fixture][plugins][plugins] logged
[00:00:01]               │ debg Logging Event for Saved Object 63cd19ee-ff34-4174-a07d-7ecd3667fd75
[00:00:01]               │ debg Logging Event for Saved Object 63cd19ee-ff34-4174-a07d-7ecd3667fd75
[00:00:01]               │ debg Logging Event for Saved Object 63cd19ee-ff34-4174-a07d-7ecd3667fd75
[00:00:01]               │ debg Logging Event for Saved Object 63cd19ee-ff34-4174-a07d-7ecd3667fd75
[00:00:01]               │ debg Logging Event for Saved Object 63cd19ee-ff34-4174-a07d-7ecd3667fd75
[00:00:01]               │ proc [kibana]   log   [03:03:00.993] [info][eventLog][plugins] event logged: {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}],"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"test 2020-04-29T03:03:00.829Z","@timestamp":"2020-04-29T03:03:00.990Z","ecs":{"version":"1.5.0"}}
[00:00:01]               │ proc [kibana]   log   [03:03:01.190] [info][eventLogFixture][event_log_fixture][plugins][plugins] test fixture: log event: 63cd19ee-ff34-4174-a07d-7ecd3667fd75 {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}]},"message":"test 2020-04-29T03:03:00.829Z"}
[00:00:01]               │ proc [kibana]   log   [03:03:01.200] [info][eventLogFixture][event_log_fixture][plugins][plugins] test fixture: log event: 63cd19ee-ff34-4174-a07d-7ecd3667fd75 {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}]},"message":"test 2020-04-29T03:03:00.830Z"}
[00:00:01]               │ proc [kibana]   log   [03:03:01.213] [info][eventLogFixture][event_log_fixture][plugins][plugins] test fixture: log event: 63cd19ee-ff34-4174-a07d-7ecd3667fd75 {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}]},"message":"test 2020-04-29T03:03:00.829Z"}
[00:00:01]               │ proc [kibana]   log   [03:03:01.244] [info][eventLogFixture][event_log_fixture][plugins][plugins] test fixture: log event: 63cd19ee-ff34-4174-a07d-7ecd3667fd75 {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}]},"message":"test 2020-04-29T03:03:00.830Z"}
[00:00:01]               │ proc [kibana]   log   [03:03:01.248] [info][eventLogFixture][event_log_fixture][plugins][plugins] found existing saved object
[00:00:01]               │ proc [kibana]   log   [03:03:01.249] [info][eventLogFixture][event_log_fixture][plugins][plugins] logged
[00:00:01]               │ proc [kibana]   log   [03:03:01.261] [info][eventLogFixture][event_log_fixture][plugins][plugins] test fixture: log event: 63cd19ee-ff34-4174-a07d-7ecd3667fd75 {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}]},"message":"test 2020-04-29T03:03:00.829Z"}
[00:00:01]               │ proc [kibana]   log   [03:03:01.265] [info][eventLogFixture][event_log_fixture][plugins][plugins] found existing saved object
[00:00:01]               │ proc [kibana]   log   [03:03:01.266] [info][eventLogFixture][event_log_fixture][plugins][plugins] logged
[00:00:01]               │ proc [kibana]   log   [03:03:01.270] [info][eventLog][plugins] event logged: {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}],"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"test 2020-04-29T03:03:00.829Z","@timestamp":"2020-04-29T03:03:01.248Z","ecs":{"version":"1.5.0"}}
[00:00:01]               │ proc [kibana]   log   [03:03:01.270] [info][eventLog][plugins] event logged: {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}],"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"test 2020-04-29T03:03:00.830Z","@timestamp":"2020-04-29T03:03:01.266Z","ecs":{"version":"1.5.0"}}
[00:00:01]               │ proc [kibana]   log   [03:03:01.277] [info][eventLogFixture][event_log_fixture][plugins][plugins] found existing saved object
[00:00:01]               │ proc [kibana]   log   [03:03:01.278] [info][eventLogFixture][event_log_fixture][plugins][plugins] logged
[00:00:01]               │ proc [kibana]   log   [03:03:01.280] [info][eventLog][plugins] event logged: {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}],"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"test 2020-04-29T03:03:00.829Z","@timestamp":"2020-04-29T03:03:01.277Z","ecs":{"version":"1.5.0"}}
[00:00:01]               │ proc [kibana]   log   [03:03:01.282] [info][eventLogFixture][event_log_fixture][plugins][plugins] found existing saved object
[00:00:01]               │ proc [kibana]   log   [03:03:01.283] [info][eventLogFixture][event_log_fixture][plugins][plugins] logged
[00:00:01]               │ proc [kibana]   log   [03:03:01.287] [info][eventLog][plugins] event logged: {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}],"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"test 2020-04-29T03:03:00.830Z","@timestamp":"2020-04-29T03:03:01.283Z","ecs":{"version":"1.5.0"}}
[00:00:01]               │ proc [kibana]   log   [03:03:01.295] [info][eventLogFixture][event_log_fixture][plugins][plugins] found existing saved object
[00:00:01]               │ proc [kibana]   log   [03:03:01.297] [info][eventLogFixture][event_log_fixture][plugins][plugins] logged
[00:00:01]               │ debg calling /api/event_log/event_log_test/63cd19ee-ff34-4174-a07d-7ecd3667fd75/_find
[00:00:01]               │ proc [kibana]   log   [03:03:01.299] [info][eventLog][plugins] event logged: {"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}],"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"test 2020-04-29T03:03:00.829Z","@timestamp":"2020-04-29T03:03:01.296Z","ecs":{"version":"1.5.0"}}
[00:00:01]               │ debg --- retry.try error: expected 1 to equal 6
[00:00:02]               │ debg calling /api/event_log/event_log_test/63cd19ee-ff34-4174-a07d-7ecd3667fd75/_find
[00:00:02]               │ debg --- retry.try failed again with the same message...
[00:00:02]               │ debg calling /api/event_log/event_log_test/63cd19ee-ff34-4174-a07d-7ecd3667fd75/_find
[00:00:02]               │ debg calling /api/event_log/event_log_test/63cd19ee-ff34-4174-a07d-7ecd3667fd75/_find?per_page=3
[00:00:03]               │ debg failed to match {"foundEvents":[{"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}],"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"test 2020-04-29T03:03:00.829Z","@timestamp":"2020-04-29T03:03:00.990Z","ecs":{"version":"1.5.0"}},{"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}],"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"test 2020-04-29T03:03:00.829Z","@timestamp":"2020-04-29T03:03:01.248Z","ecs":{"version":"1.5.0"}},{"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}],"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"test 2020-04-29T03:03:00.830Z","@timestamp":"2020-04-29T03:03:01.266Z","ecs":{"version":"1.5.0"}}],"expectedEvents":[{"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}]},"message":"test 2020-04-29T03:03:00.829Z"},{"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}]},"message":"test 2020-04-29T03:03:00.829Z"},{"event":{"provider":"event_log_fixture","action":"test"},"kibana":{"saved_objects":[{"namespace":"default","type":"event_log_test","id":"63cd19ee-ff34-4174-a07d-7ecd3667fd75"}]},"message":"test 2020-04-29T03:03:00.829Z"}]}
[00:00:03]               └- ✖ fail: "event_log Event Log public API should support pagination for events"
[00:00:03]               │

Stack Trace

{ Error: expected 'test 2020-04-29T03:03:00.830Z' to sort of equal 'test 2020-04-29T03:03:00.829Z'
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
    at forEach (test/plugin_api_integration/test_suites/event_log/public_api_integration.ts:179:40)
    at Array.forEach (<anonymous>)
    at assertEventsFromApiMatchCreatedEvents (test/plugin_api_integration/test_suites/event_log/public_api_integration.ts:176:19)
    at Context.it (test/plugin_api_integration/test_suites/event_log/public_api_integration.ts:70:7)
  actual: 'test 2020-04-29T03:03:00.830Z',
  expected: 'test 2020-04-29T03:03:00.829Z',
  showDiff: true }

---

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream