[FEATURE] Refactor integration setup process

[Swiddis]

Is your feature request related to a problem?

A critical part of the integration building process is setting up external assets, which define the ways that integrations can read and process data. Currently, assets are stored as references in the config, split by the type of asset (today: saved objects and queries).

As part of the setup process, we go through all of these assets and follow hardcoded steps to install each category of assets. Queries are sent to the asynchronous Flint API, saved objects are sent through a high-level client in-code. This has worked reasonably well so far, as it makes adding more asset types as easy as adding a definition to the assets field, and integrating it with the existing setup process.

But with growing feature demand, this approach is proving insufficient. There are issues like #1442 demanding being able to set up assets associated with arbitrary OpenSearch APIs, and in-progress features for supporting multiple installation flows¹ within one integration that would require configuring which assets are installed. Furthermore, splitting the assets into distinct entities like this complicates serialization and bundling. A different approach is needed.

What solution would you like?

Currently, the assets field of integrations looks like this:

"assets": {
  "savedObjects": {
    "name": "dashboards",
    "version": "1.0.0"
  },
  "queries": [
    {
      "name": "create_table",
      "version": "1.0.0",
      "language": "sql"
    },
    {
      "name": "create_mv",
      "version": "1.0.0",
      "language": "sql"
    }
  ]
},

I propose updating the format to look like this:

"assets": [
  {
    "name": "create_table",
    "version": "1.0.0",
    "extension": "sql",
    "type": "query"
  },
  {
    "name": "create_mv",
    "version": "1.0.0",
    "extension": "sql",
    "type": "query"
  },
  {
    "name": "dashboards",
    "version": "1.0.0",
    "extension": "ndjson",
    "type": "savedObjectBundle"
  }
],

By unifying the categories, serialization switches from handling each case to writing one main handler. This also allows more flexible installation ordering: if there are queries to run after installing a saved object bundle, that's trivially addable. The install logic will remain case-by-case, but it's somewhat unavoidable if different objects require fundamentally different interfaces to install, and it lets us rework the type-by-type logic to something like for (const asset of assets) { install(asset); }².

One concern is backwards compatibility: since dynamic catalogs was released with the old serialization, at least the OS integrations plugin should be robust to both versions. This can be added on-read by having a validation check which automatically converts the old format to the new format, but there are still some details with how to make migrations work with e.g. the .kibana type mapping.

What alternatives have you considered?

We can continue adding case-by-case steps, which has the drawbacks mentioned above, and is better for backwards compatibility. To implement ordering we could add a field to each type like order, where steps run sorted by order (default: 0, and support negative values). I worry this will continue to get worse as we add more asset types, but is minimal work upfront.

We could also provide some sort of helper method which takes the current format and converts it to this new format. That method can be relatively robust, and simplify a lot of operations on the asset list without requiring a change to the serialization format.

Do you have any additional context?

Also related: workflow integrations with opensearch-project/observability#1805. Additionally, the already-cited request for arbitrary API calls #1442.

Footnotes

1. To support distinct installation paths, a further flows?: string[] field can be introduced (default: all flows), which specifies when a step should be run. If dashboards in the above assets are only installed for a full installation, a user can add the entry "flows": ["full"]. There may need to be more work to determine how to fully support installation flows, I'll leave that for another issue. ↩

2. A sophisticated implementation might include dependency logic and run steps in parallel, but this is unlikely to be necessary for the amount of effort it would add. ↩

[YANG-DB]

Nice 👍

1. How would the actual asset define the API call - for example I would like to call the _reindex API as part of the Integration's elements - can u plz give an example for these type of assets ?
2. How would the order of assets be installed ? can they be depended on one another - meaning if one failed the other will not attempt ?

[Swiddis]

1. Not sure which is better: adding extra fields under certain types of assets that let you add that info ("url": "...", "body": "...", "method": "..."), or putting the request in another file with a well-defined format (e.g. Hurl) and loading+running that. My instinct is to use the file solution -- adding a library to handle parsing it increases our dependency footprint but we get a lot of functionality in exchange.
2. I'm thinking we should just start by running the list top-to-bottom where any failure cancels the installation. In a footnote I mention running steps in parallel, but it seems like premature optimization at this stage. (How to/whether to rollback steps is still an open question...)