[AWS] Add support for granular search of cloud service integrations

[sorantis]

Today when users search for AWS on Integrations page in Kibana they see the following result

Searching for any specific AWS service however doesn't yield any results. For example searching for aws ec2, ec2 looks like this

In contrast, the marketing integrations page already provides service level granularity for AWS integration - when users search EC2 they can see a tile for it.

User Stories

As an integrations user, I want to search for individual services within a larger platform, and add them independently of the other services.

As an integrations user, I want to search for keywords (e.g. database) or browse an existing integration category (e.g. Datastore) and see AWS services pertaining to my search criteria.

The AWS package today doesn't not provide any granular information about what services it can monitor. Because of that there's a 1:1 mapping between a package and a corresponding tile. The goal is to show a tile for each service that the AWS package can monitor. To achieve that the package should publish information about each service integration, i.e. a manifest.yml file containing information about each service, such as service name, description, icon, keywords, etc.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[kaiyan-sheng]

+1 Instead of exposing AWS at the package level, it would be beneficial to expose everything at the dataset level. One question: with the granular integrations, when users search for AWS EC2, does EC2 logs and EC2 metrics show up as two separate integrations or they would be under one tile called EC2 logs and metrics?

[sorantis]

I think the latter, the logs and metrics section should be shown after the user has clicked the AWS EC2 tile with respective details.

[tbragin]

cc @mostlyjason @mukeshelastic Team - I assume we're doing this consistently across all cloud integrations, not just AWS?

[sorantis]

Yes, AWS is the first cloud integration we're considering. Other cloud integrations will follow the same pattern (where applicable).

[mostlyjason]

@sorantis I assume this issue is a placeholder for us to define the requirements and patterns in more detail? I imagine we'll have a proposal of some kind and then ask for the team to review it once we are ready? There are some questions it'd be good to define like how it applies to other platforms, how it works in the UI, where credentials are stored, etc. We can sync and start mapping it out in more detail.

[sorantis]

@mostlyjason the scope of this issue to only cover service tiles for cloud integrations on the example of AWS. I updated the title to reflect that. As the user stories suggest the change should only impact the package itself, and have minimal to none UI impact. @masci and @kaiyan-sheng will prepare a technical proposal document for review.
The second part, the actual integration configuration workflow for AWS (and other cloud services) will be covered in a separate issue, with mockups, design doc where questions like impact on the UI, credentials and more will be covered. The initial conversation has already been scheduled with you.

[mostlyjason]

should only impact the package itself, and have minimal to none UI impact

@sorantis we talked about splitting logs and metrics into separate integrations within the AWS package. I think the UI doesn't have a way to support multiple integrations per package today. Are you tracking that here or elsewhere?

[kaiyan-sheng]

I started with trying to figure out what we can do in our integrations package to achieve the granular search. But without duplicating the integration packages, it doesn't seem to be doable. Please see #577 for more details.

Going back to the original problem and trying to solve it without duplicating packages, @ycombinator and I had a discussion. In the current Kibana Fleet/Integrations page, we are only displaying tiles at the package level. If Kibana can expose integration tiles at the data stream level for certain integrations(like aws in this case), that would solve this problem.

Right now the icons for integrations are defined in manifest.yml file at the package level. We can add a config option(For example: expand_data_stream: true) to indicate if Kibana should display all data streams underneath in Fleet/Integrations. If expand_data_stream is set to true under packages/aws/manifest.yml, then Kibana should display all data streams under aws package as separate integrations. We can introduce a new icons section in manifest.yml file under each data_stream for Kibana to display the tiles. For example, we can add the AWS EC2 icon under packages/aws/data_stream/ec2_metrics/manifest.yml and this icon would be used in the Kibana Fleet/Integrations page.

For documentations, we can have a separateREADME.md file under each data stream and that will be presented on the Kibana UI when the user clicks a data stream tile such as AWS EC2. But when the user clicks Add AWS EC2 icon, the same configuration as the AWS package should show up. AWS EC2 logs and AWS EC2 metrics should be enabled by default because user clicked the Add AWS EC2 icon. The rest of the services should be disabled by default but still show up in the config.

Sample integrations/packages/aws/manifest.yml file:

policy_templates:
  - name: aws
    title: AWS logs and metrics
    description: Collect logs and metrics from AWS services
    expand_data_streams: true
    inputs:
      - type: s3
         ...
      - type: aws/metrics
         ....

Sample integrations/packages/aws/data_stream/ec2_metrics/manifest.yml file:

title: AWS EC2 metrics
release: beta
type: metrics
icons:
  - src: /img/logo_aws_ec2.svg
    title: logo aws ec2
    size: 32x32
    type: image/svg+xml

[ycombinator]

Thanks for summarizing our conversation here @kaiyan-sheng.

Creating per-datastream packages that duplicate information from the parent integration package (as explored in #577) is not a great design because it can lead to drift in the contents between the parent and child (per-datastream) packages. We also discussed the idea of "lightweight packages" that would simply contain pointers to specific datastreams in a parent package but even this is overkill IMO since the package is already structured into data streams and the UI is already aware of this structure (as seen in the integration configuration page in the bottom half).

What we are missing are some metadata fields at the datastream level in the package spec and UI enhancements to take advantage of these new metadata fields.

Package spec enhancements

These are detailed in the previous comment so I won't repeat them here. I will just add that the proposed enhancements jive quite well with the requirements mentioned in this issue's description:

The goal is to show a tile for each service that the AWS package can monitor. To achieve that the package should publish information about each service integration, i.e. a manifest.yml file containing information about each service, such as service name, description, icon, keywords, etc.

We already have manifest.yml files at the datastream level. These can be enhanced (as detailed in the previous comment) to contain fields for description, icon, keywords, README, etc. as needed by the search results tile and integration/datastream detail page.

UI enhancements

• showing the data-stream level icons along side packages on the search results page for packages that match the search terms and have expand_data_streams (or similar) set to true in their manifests.
• allowing the integration detail page to take an optional data stream parameter that shows details about that specific data stream from the package OR creating a new data stream detail page if that's better (defer to the UI devs on this).
• allowing the integration configuration page to take an optional data stream parameter that will enable only that data stream (or data streams) and disable the others by default.

[kaiyan-sheng]

@nchaulet and @skh What do you think from the UI side?

[skh]

For the design / UX part I'd like to defer this question to @hbharding .

Currently, the search is not very sophisticated, we more or less take the response from the package registry's /search endpoint (either all, or by category) and filter it down client-side. That means we do not know anything about the datasets in this view, and enhancing the feature might mean enhancing the functionality of the registry's /search endpoint.

An example what the data looks for us right now is https://epr.elastic.co/search?experimental=true

[mtojek]

// silently suggests that this is a good candidate for a thread in the package-spec repo

[kaiyan-sheng]

As @mtojek suggested, moving this thread into package-spec repo: elastic/package-spec#122

[sorantis]

Closing as a duplicate elastic/kibana#93315