agent cloudbeat support

[oren-zohar]

What does this PR do?

Adds cloudbeat support to the agent.

Why is it important?

As part of the Cloud Security Posture team work to add our own beat to collect cloud posture information and send it to ES, we need to add the relevant configuration changes in order to support it in the agent.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Relates cloudbeat libbeat changes beats#30699

[mergify]

This pull request does not have a backport label. Could you fix it @oren-zohar? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-03-29T15:08:46.753+0000

• Duration: 39 min 12 sec

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages.

• run integration tests : Run the Elastic Agent Integration tests.

• run end-to-end tests : Generate the packages and run the E2E Tests.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[ph]

/test

[ph]

Small changes around the initial arguments.

[ph]

You can remove this, I will double-check the other existing spec,

[ph]

Any reason why you only support Elasticsearch?, since you are using the libbeat framework you should have access to all of the outputs? see https://github.com/elastic/elastic-agent/blob/main/internal/spec/apm-server.yml#L37

[ph]

This look good to me, but @narph Would having their packages added into our dev-tool yaml definition would make our build fail until the artifacts exits?

[ph]

@oren-zohar The changes for the spec files lgtm, the main branch is the 8.2 release are you planning to release for 8.2?

[oren-zohar]

@ph yes I think we should merge it. Everything we work on should be hidden behind a feature flag, so it will not affect the release AFAIK.

[narph]

This look good to me, but @narph Would having their packages added into our dev-tool yaml definition would make our build fail until the artifacts exits?

It might cause some issues, maybe the artifacts can come first, I am not sure what is the progress on that.

@ph , shouldn't introducing a new spec cause a regeneration of the supported.go file? @oren-zohar have you ran mage buildspec?

[oren-zohar]

It might cause some issues, maybe the artifacts can come first, I am not sure what is the progress on that.

That makes sense, @DaveSys911 wdyt?

@ph , shouldn't introducing a new spec cause a regeneration of the supported.go file? @oren-zohar have you ran mage buildspec?

@narph I was thinking the same, we did run the buildspec script but it didn't change supported.go for some reason.

[eyalkraft]

@oren-zohar , @yashtewari says pulling from main will fix the buildspec script

[oren-zohar]

@narph @ph added the missing part in supported.go e2e fails but I'm not sure if it's related (also fails in main)

[ph]

Just to clarify what I've said on slack, let me clarify the elastic agent behavior.

1. When an input is present in the Elastic Agent.
2. The agent looks on the disk if the archive is present.
3. When the archive is not present it will fetch it over the artifact api.

When the Elastic Agent is built from the release manager, the manager is responsible for building everything independently and assembling the distribution. It makes sure that the artifacts is present inside the archive or the docker container.

Now, the files in dev-tools are required only in development when we are trying to build Elastic Agent locally, it's for an ease of development, I believe this is not required for cloudbeat and that would add an unnecessary link between our code.

Only the changes in the spec files are needed.

[ph]

Change lgtm, waiting for ci to complete to merge.

[ph]

Merged, I've reworded the commit message.