[Fleet] Integration testing with the registry

[jfsiii]

The Integrations Manager will have a client which communicates with the registry to fetch integration metadata and artifacts. With the Manager and Registry so tightly coupled, there should be some level of integration testing.

Since the Manager is the only way Kibana can install/remove integrations, it's incredibly important they stay in sync.

The registry publishes example responses which the Manager can use to confirm certain features. APM does something similar to watch their interface with ES.

Another option is for the Registry to publish machine readable info using a common format like JSON Schema or Open API (fka Swagger). I'm most familiar with Open API but I'd like to discuss either/both these options because they give us greater control, durability, and tooling.

There's a quick read with a good overview at https://apisyouwonthate.com/blog/json-api-openapi-and-json-schema-working-in-harmony

[jfsiii]

https://spin.atomicobject.com/2018/03/26/typescript-data-validation/ has some more concrete examples/comparisons.

Coming back to the tooling advantages of JSON schema, if the registry used a golang package like https://github.com/mcuadros/go-jsonschema-generator to publish a JSON schema, the Manager could use an npm package like https://github.com/bcherny/json-schema-to-typescript to convert those to TypeScript interfaces

[ruflin]

@kuisathaverat @mdelapenya I wonder if this is an effort one of you could help with.

[mdelapenya]

@ruflin just to understand the main goal here: the Integrations Manager's client must follow a contract, and we should check that the client adheres to that contract, right?

[ruflin]

I think there are few levels to this:

• Have actual integration tests running the registry and Kibana in parallel and execute some tasks to see if the it works as expected
• Have a simple way in Kibana during development and unit testing to confirm that the code changes still work with the data from the registry.
• Have a contract between the two expose as a schema and very in both test suites, that we still conform to it.

Long term I think we should do all three but the easiest one to get in would be 2. @jfsiii has also above a few proposals we could look at which can also have a positive effect on development.

I think especially for 1 we would need some help to set it up but I don't see this as urgent right now.

Let's continue using this issue to discussing what are the approaches we want to take and what tools we should use for it. I'm sure I also missed some more options in the list above.

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[ruflin]

@nchaulet This is related to running the docker registry for testing.

[jen-huang]

I don't think this issue is relevant any more, we have been using dockerized package registry in our integration tests for a long time now.