Separate AWS services into packages

[kaiyan-sheng]

Conclusion

This approach does not work.

What does this PR do?

This PR is to separate AWS services into their own packages to have granular support when searching cloud services. With this PR, when searching for
AWS on integrations page, both AWS and AWS EC2 will show up:

When searching for ec2, AWS EC2 integration will show up:

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all datasets collect metrics or logs.

Related issues

• Relates [AWS] Add support for granular search of cloud service integrations #526

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Branch indexing

• • Start Time: 2021-01-29T22:28:45.882+0000

• Duration: 38 min 37 sec

• Commit: 84ea615

Test stats 🧪

Test	Results
Failed	0
Passed	1641
Skipped	0
Total	1641

[kaiyan-sheng]

@sorantis With this approach, when user wants to add EC2 integration and click the AWS EC2 tile, it does not bring the user to the original AWS integration page. This page is dedicated to EC2 related logs and metrics. Here is what the AWS EC2 integration looks like (still needs clean up):

Is this OK or we should actually take the user back to the full AWS package integration page once the user clicks the EC2 tile?

[kaiyan-sheng]

OK @sorantis @masci and I had a meeting earlier and decided this is not fully what we are looking for.

Two parts here:

1. Front end: When user goes to integrations and search for aws, all tiles related to AWS should show up. When user goes to integrations and search for ec2, the AWS EC2 tile should show up. This PR does achieve what we want here.
2. Back end: We do not want to have aws_ec2 as a separate package because now we need to maintain two packages: aws and aws_ec2. Also what we are looking for is: when the user clicks to add AWS EC2 integration, the same list of services(logs/metrics) show up as if user clicked to add AWS integration. Only difference should be when user clicked through AWS EC2, the EC2 logs and metrics should be enabled by default and the rest are disabled.

[kaiyan-sheng]

With the new commit, we can see when the user clicked add AWS EC2, the same config page (same as clicking AWS) show up but only EC2 metrics and EC2 logs are enabled by default. Similar to DynamoDB, only DynamoDB metrics is enabled by default.

Problem: this commit still requires duplicating the integration packages.

[kaiyan-sheng]

Update: I can't find a way to manipulate manifest.yml file to achieve what we want without duplicating packages for each service.

With duplicated packages like this, maintaining all of them in the future will be problematic.