Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Testing for integrations in CI #74

Open
Swiddis opened this issue Aug 21, 2023 · 0 comments
Open

[FEATURE] Testing for integrations in CI #74

Swiddis opened this issue Aug 21, 2023 · 0 comments
Labels
cli command line interface enhancement New feature or request

Comments

@Swiddis
Copy link
Collaborator

Swiddis commented Aug 21, 2023

Is your feature request related to a problem?
We currently don't have very good CI for validating whether integrations work. There are a few tests that I historically have manually tried to do every time I want to check an integration, which often are missed by devs and increase dev time, and also have been historically missed in bug bashes (e.g. many current 2.9 integrations fail check (4)):

  1. Check the config for any clear issues, make sure it loads correctly*.
  2. Move the integration to a local directory and run osints diff to find type errors.
  3. Try to create the integration manually with "try it" and ensure that panels aren't erroring.
  4. Try to connect to the integration's own sample index to ensure that there's no dormant type mismatches caused during index creation.
  5. Checks for cross-interference: Creating multiple copies of integrations, creating sequences of integrations, trying to make joins, all that fun stuff. These are more complicated to test and should really be looked at after (1)-(4) are met.

* - This step has been automated in the current plugin CI, but will need to be moved to this repository when we do the catalog move.

What solution would you like?
At a glance, all of these tests except maybe (3) and (5) seem simple to automate. We should have a CI pipeline, ideally at a unit test level, which can perform all of these on the catalog. That said, there is a wider issue of how we actually want to introduce the CI pipeline to this repository. For example, there are existing tests in the plugin written in Typescript, while the CLI code here is written in Python, and the Typescript tests depend on plugin methods to run.

At a glance, my instinct is to have one small test action on this repo that does unit testing to validate integrations for (1) and (2), and a larger "soak test" action that does a full integration test with the integrations plugin for (3), (4), and (5).

What alternatives have you considered?

  • Rewrite Typescript tests in Python, add a CI pipeline to the CLI that includes validating any catalog that we put here.
  • Copy CI over with new validation methods and separately run Python and Typescript CI, which might duplicate a lot of code.
  • We could also look at installing the Integrations plugin within the CI, but this will also mean inheriting dashboards-observability's long build times for only a few methods.
  • Convert CLI logic to Typescript and somehow merge it with the Integrations plugin, saving duplication but still having the issue of tests and content being in different locations.

Do you have any additional context?

@Swiddis Swiddis added enhancement New feature or request untriaged labels Aug 21, 2023
@Swiddis Swiddis changed the title [FEATURE] Testing for integrations in CI. [FEATURE] Testing for integrations in CI Aug 21, 2023
@YANG-DB YANG-DB removed the untriaged label Sep 7, 2023
@YANG-DB YANG-DB added the cli command line interface label Aug 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cli command line interface enhancement New feature or request
Projects
Status: No status
Status: No status
Development

No branches or pull requests

2 participants