Support granular search of AWS services #122
Comments
kaiyan-sheng
added
Team:Integrations
|
@kaiyan-sheng I reviewed the draft PR (elastic/package-registry#681) to follow the changes you're proposing. I think you should start with a PR to the package-spec suggesting modifications to the package spec (I noticed that you'd like to introduce icons/screenshots per data streams and |
|
I'm not sure if the info you are interested in is actually coming from the data stream level. Lets take EC2 above: There are at least 2 data streams here, one for logs and one for metrics. Instead I think each integration (like EC2) is an additional entry in the |
|
I had a "short" call with @ycombinator today regarding enabling this feature. Let's call them temporarily - promoted data streams. I tried to prepare a sample test package with promoted data streams. It looks like like promotion requires extra elements to be present like icons (with new directory), READMEs (with new directory), READMEs templates. Not sure if all data stream READMEs should be glued together into a package README or do we want to keep them consistent. You can consider this sample package as an input for tests for this feature: https://github.com/mtojek/package-spec/tree/promote-data-stream-tests/code/go/internal/validator/test/packages/promoted_data_streams A data stream can be highlighted as promoted with a manifest flag: I'm not sure about description and title properties, which are not part of the data stream, but part of every title: Second data stream
type: logs
streams:
- input: logfile
title: Second data stream title
description: Second data stream description
icons:
- src: /img/first_icon.svg
title: Second data stream icon
size: 32x32
type: image/svg+xml
promoted: true |
|
I'm not sure these are streams or policy templates. I agree with @ruflin that these are not streams since there are multiple streams per service, and we only want to show services on the search page. That means that the streams for EC2 logs and metrics should create a single tile for EC2. I'm not sure they are policy templates either because @sorantis proposed that we'd have only two policies for AWS, one for logs and one for metrics. This means the policy template alone cannot group by service. I think we need a way for each integration to provide a list of services as metadata at the top of the integration. Each service will have a title, description, icon, etc. The service tiles are just links back to the parent integration. They are virtual integrations in the sense that they don't exist on their own, they just provide easily searched shortcuts back the parent. This issue has nothing to do with how the integration policy editor is built or displayed. I think we should open a separate issue for how to specify which service each stream is assigned to, and how that builds the policy editor. |
@ruflin Good point! We do want EC2 to be displayed as one icon instead of two (one for metrics, one for logs). Can we add a separate section called promoted_data_streams in the top-level |
|
Thanks @mostlyjason, your comment simplifies a few things, in particular:
This was not the requirement I was made aware of; instead my understanding was that clicking on the service tiles was supposed to take the user to a service detail page which would be similar to the integration detail page in layout but contain service information instead. Additionally, I was told clicking on the "Add Integration" button from that page should take the user to the parent integration configuration page but with only the selected service's data streams enabled. But given what you've just said, we can design something much simpler. I'm imagining an optional services:
- name: ec2
title: EC2
description: ...
icons:
- src: /img/logo_aws_ec2.svg
title: AWS EC2 logo
size: 32x32
type: image/svg+xml
- name: s3
title: S3
...
...When the package registry starts up and builds an index of packages, it could parse the |
Out of curiosity, did we consider introducing a new integration called just "EC2 Integration" that combine logs and metrics instead? |
|
@mtojek We did consider that in elastic/integrations#577 (comment). With this design, we will have to maintain all these individual service integrations, which will be a nightmare (versioning and etc). Also when users want to enable multiple services under AWS, they have to duplicate the onboarding process multiple times, which is not user friendly. cc @masci |
|
@mostlyjason @sorantis When a user clicks on the AWS EC2 icon and clicks add integration, the same page for adding AWS integration should show up but with only EC2 metrics and EC2 logs enabled by default, correct? Will Kibana have a way to identify EC2 metrics and EC2 logs that should be enabled or this requires changes in EPR as well? Maybe we need a |
|
@jen-huang It would be good that you keep on eye on this, this will require probably changes in our UI. |
Yes that sounds right. I was thinking of the search page as a separate issue from the integration detail and integration policy editor pages. They are all related to the same user journey and need to link to each other. So based on which tile the user selects on the search page, the integration detail could either be specific to each service, or have a section that is activated for each service. I don't remember seeing a mockup from @sorantis on what the detail page would show. It'd be nice to know what that looks like though. @hbharding and I can help with the nginx use case if desired. +1 on having a services section, and associating the services with streams. |
|
Thanks @mostlyjason. @kaiyan-sheng Based on Jason's comment above, I think what you proposed in #122 (comment) is good. The only tweaks I'd suggest are:
I think there's sufficient consensus at this point to move forward with a PR to the package spec. We can iterate on further changes once you have the PR in review. Thanks! |
|
@ycombinator Thank you for the suggestions! Please let me know if the section below looks better now! I can start a PR in package spec and discuss more there! |
|
@kaiyan-sheng Yup, LGTM. PR time! |
|
The discussion internally is still ongoing on what the exact concept should be so it is impossible to map to a spec yet. Lets hold this discussion until we all agreed on the mocks and the user flow. |
I thought we had agreed that the At the very least it would be helpful if discussions happening in other issues were linked from here or vice versa (in case of private repo issues). And if discussions are happening in Zoom, Slack, etc. — that's fine but please bring them back to issues. Otherwise its very frustrating to those of us not privy to all the discussion and not a very open way of collaboration. 😞 |
|
+1 on holding off development until we agree on the definition. @sorantis is leading an effort to define this initiative in more detail and it is still WiP. We need approval from drivers and key stakeholders before proceeding. |
|
++ in the future lets not file GH issues until definitions/design have settled elsewhere. Otherwise it leads to confusion and churn like we saw here. |
|
Closing this issue since PRs on both integrations side and Kibana side are merged. |
Introduction
Instead of only exposing the AWS package in Fleet/Integrations list, all supported services inside AWS package should also have their own icons and displayed in Fleet/Integrations list in Kibana UI. Using EC2 as an example, under Integrations, we should have both AWS and EC2 icons:
Design
Here is my initial design. I split them into three parts for three repos I believe we need to make changes on: elastic/integrations, elastic/package-registry, and elastic/kibana.
Part 1(integration):
In
elastic/integrationsrepo, for packages, we should introduce two new config options intomanifest.ymlfiles.At the integration package level, we have a
manifest.ymlfile that includes the icon that UI is using for the whole integration. We should add a config parameterexpand_data_streamat the package level to indicate this package should be expanded to display all data streams.For example,
manifest.ymlshould look like this:At the data stream level, we also have an existing
manifest.ymlfile. We should add a section foriconsand a section fordescriptionfor each data stream so whenexpand_data_streamis true, these icons at the data stream level will be displayed in Kibana Fleet/Integrations list.For example,
manifest.ymlfile should look like this:Part 2(Elastic Package Registry):
In the Elastic Package Registry(EPR), we should adjust the code to read the new config option
expand_data_stream. Whenexpand_data_streamis set totrue, abstract data stream info and store them into the/searchAPI endpoint result. I created a draft PR for this to show the changes to the/searchAPI response.For example with
expand_data_stream: truein AWS package,/searchAPI response will look like this:Part 3(Kibana Fleet UI):
Continue to take the response from the EPR's
/searchendpoint, also take into account theiconssection underbase_data_streams.Follow up
Once we achieve the granular search for AWS, we can split
README.mdinto each data stream/service. This way when the user clicks on an icon for a specific service, some dedicated documentation will show up instead of the full documentation.@hbharding @jasonrhodes WDYT about this approach?
cc @sorantis @masci
The text was updated successfully, but these errors were encountered: