[Stack Monitoring] Testing strategy for agent/integration data

[neptunian]

Aside from manual testing we need to write functional and api integration tests that use the new metrics-* agent data for each integration. There are a few ways to go about it

• @chrisronline locally modified the es_archiver to take the existing .monitoring archive data and rebuild it into metricbeat data. See Part 2. This means he was able to copy over all the existing functional and api integration tests without having to change anything except the archive that was to be loaded. We could do something similar and convert the existing metricbeat-* data to metrics-* data. Since we never actually supported metricbeat-* or ensured the UI worked with it, I don't know if this data is totally correct.
• Generate new data. This would mean we'd need the es_archiver to support us saving new data, which it does not yet fully Add support for data streams in ES Archiver #69061 . It would also mean going over our tests and creating new tests and have them match this new data. It would also provide us an opportunity to make sure the data is correct and understand the app better. I am leaning towards this.
• @klacabane Suggested using live data in the tests but I think this would probably add too much time and flakiness

@klacabane and I are going to investigate the best approach, see what others have done, and see how difficult it would be use/modify the es_archiver to work with data streams.

** Update **

After some discussion with @klacabane, we decided the following:

• Use the package registry docker image to load actual packages so we can have the latest mappings. We can pass in a custom configuration yaml that points to a server and install the latest package with the endpoint /api/fleet/epm/packages/${pkg}/ which should install the latest version
• generate agent/integration data by transform the existing _mb archived data, adding the correct datastream index value or any other values that are necessary (using something similar to this script)

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[matschaffer]

I like parts of both the first two bullet points. Being able to run the same test code over two data sets sounds like a nice move.

But also if the new code will be data streams, it seems like the test should be loaded as a data stream too. Of course, it might not matter much.

The UI will query metrics-* which could match indices or data streams and I wouldn't expect the query responses to change.

[miltonhultgren]

I know we're still early on this but @matschaffer and I are looking into how to simulate Stack Monitoring data with apm-synthtrace, which supports writing to data streams. Maybe it's an option to consider even though it would require a bit more investment in our data generation tooling but I think we're on that path anyway and this would be for a concrete problem.

[klacabane]

I'm wondering if we currently have validation of the mappings, or if that's even necessary with our versioning model.
In an agent world the mappings are defined by the Integrations package and we have a frontend component that expects a specific mapping definition to query and present data. What if the mapping changes, for example removing a property. Would that be guaranteed to be caught by end-to-end tests (or another mechanism), or are there scenarios where it could go unnoticed ?

[matschaffer]

are there scenarios where it could go unnoticed ?

I'm fairly certain the answer to that is "yes". It's sounds very close to the failure scenario I demoed during the nov 15th team meeting though granted, in your scenario the mappings come from packages rather than ES itself.

[klacabane]

I guess we could benefit from contract/schema tests on the mappings. It will be valuable to initially validate an integration package mappings against our expectations, but I don't know how doable it is currently - maybe we can leverage type definitions as a source of truth for monitoring expectations. Parsing our queries would be another (complex) solution.

Regarding the strategy we could start with option 1 which sounds cheap to implement and would provide an initial coverage that quickly surface bugs in the datastream usage. Once datastream support is added to esarchiver (or we have an alternative to load data in stream), we could work on the second option

[klacabane]

So it looks like we could use esArchiver if data streams are already created (see #68794). In our case the datastreams and relevant assets are installed by the fleet application on request. I'm thinking about hitting that endpoint during the test suite setup - while it will add coupling and complexity to our testing environment (we'll need a package registry running in a docker env?), I like the idea of a tighter integrations with the packages assets, comparing to defining static mappings that could go out-of-sync.

With datastreams available, we can use esArchiver to insert the archives already defined. Transforming metricbeat-* into metrics-* documents should be straightforward and serve as a first step. As a follow up we can replace the esArchiver with another data-generation solution.

[matschaffer]

we'll need a package registry running in a docker env

I don't think anything in the current yarn-powered testing requires docker yet. Might be worth finding out what it'd take to make something like https://github.com/elastic/kibana/blob/main/packages/kbn-es/README.md for the package registry. I forget who to talk to about this, but I remember @ycombinator presenting about it at the last GAH so maybe he knows.

[ycombinator]

I'm not sure what the state of the art is with the package registry and package tooling. But I suspect @mtojek and/or @jsoriano might be able to provide the necessary guidance here.

[mtojek]

FYI Package Registry/Storage will be under active development soon. We'd like to replace the Git repository with true object storage.

Regarding testing, tools we have can help with policy- and agent-oriented testing, but they don't include any Selenium or Kibana UI tests. They just depend on the available Fleet API. Keep in mind that we don't exercise Kibana, but Elastic packages. You can find more information here, especially system tests.

[miltonhultgren]

@dgieselaar Can you share some insight on how the APM team handles this since you also need to launch APM server in your tests? (maybe you don't do it for E2E tests?)

[klacabane]

@matschaffer this is already supported but I need to verify if we actually need this or not

I was able to get a subset of elasticsearch functional tests running and succeeding against datastreams with the esArchiver approach:

• install elasticsearch integration assets (index templates..) through the fleet api
• transform metricbeat-* archived data into metrics-* data and remove mappings.json from the archive
• update esArchiver to load new datastream data with { useCreate: true } option
• update test_user permissions to allow access to datastream

[dgieselaar]

@dgieselaar Can you share some insight on how the APM team handles this since you also need to launch APM server in your tests?

We use an empty esArchive that only contains mappings/ empty indices for APM data. in the future we'll probably install the APM integration package. We don't use APM Server with Synthtrace.

Separately we might have integration tests that spin up the full APM stack (Kibana, ES, APM Server, APM Agents), but I have not worked with those before.

[matschaffer]

@matschaffer this is already supported but I need to verify if we actually need this or not

Oh, wow. Nice find! Seems like it's main usage is for the package registry, so seems like you're on the right track.

[miltonhultgren]

This issue is related: #123345 (action point from our team meeting on data generation tooling)

[neptunian]

After discussing with @klacabane we decided:

• Use the package registry docker image to load actual packages so we can have the latest mappings. We can pass in a custom configuration yaml that points to a server and install the latest package with the endpoint /api/fleet/epm/packages/${pkg}/ which should install the latest version
• generate agent/integration data by transform the existing _mb archived data, adding the correct datastream index value or any other values that are necessary (using something similar to this script)
• copy over the tests, similar to _mb, we'll have something like _agent copies

We can start with elasticsearch #119109

[matschaffer]

Is this maybe a good opportunity to try to work out a way to do testing without copying the tests themselves?

I'm thinking maybe something like a helper that sets up the same test for different setup methods.

[jsoriano]

• Use the package registry docker image to load actual packages so we can have the latest mappings.

If the only reason to start the registry is to load packages, take into account that there may be additional methods in the near future that don't require a registry, such as #122297 or #70582.

[neptunian]

• Use the package registry docker image to load actual packages so we can have the latest mappings.

If the only reason to start the registry is to load packages, take into account that there may be additional methods in the near future that don't require a registry, such as #122297 or #70582.

Thanks @jsoriano . Yes, that would be our only reason. In #122297, it looks like packages that are being shipped with Kibana are being automatically installed during Kibana setup. Is this going to change to allow packages to be shipped that aren't automatically installed? Also, are our packages considered "stack-aligned"? As in we don't need the ability to upgrade our packages out-of-band from stack releases (to fix a bug for instance) and want to make sure our packages upgrade with Kibana ("for example new features in the UI that depend on new fields that were added in the same Stack version") @sayden

[klacabane]

Rethinking this we can skip the functional tests and only focus on api tests: we already have functional (e2e) coverage of the .monitoring-* data, so if we can validate that API responses are similar when reading from metrics-* we have the guarantee that UI will respond similarly with both data sources. This will save significant effort and computing power.

The following steps will setup an initial coverage:

1. Transform .monitoring-* data into metrics-* data. Here's a script that transforms the data https://gist.github.com/klacabane/f2ef21b0c2722f312f2d983d9870dc68:
   • it adds the field used by esArchiver to detect whether the target is a datastream or an indice
   • it adds the data_stream object that replaces metricset construct for query filtering
2. Extract metrics-* mappings with esArchiver and bundle them with the output of 1.
3. Create a copy of the _mb tests that loads the bundle created in 2. and make the same assertions

As a follow up we can replace the static mappings created in 2. by installing the packages as a setup step to the test suite. This will require spawning a local package registry and hitting the fleet api. This step will allow easier updates to the mappings so that we have continuous integration in the SM tests since it will only require an update of the packages (we'll have to bundle the packages in the kibana repo to avoid any network reliance)

[klacabane]

Closing this as initial test coverage is merged. Follow up in #146000