Elastic Agent relies on artifacts-api for fetching the dependent artifacts when running the packaging

[v1v]

What

As you have probably found, since yesterday the main branch is failing when running the packaging, since artifacts are not yet available for metricbeat, filebeat and so on.

[2022-09-22T10:04:24.060Z] time="2022-09-22T10:04:24Z" level=warning msg="The Elastic artifacts API is not available yet" artifact=filebeat artifactName=filebeat-8.6.0-SNAPSHOT-linux-arm64.tar.gz elapsedTime=268.234246ms error="GET request failed with 404" retry=1 statusEndpoint="https://artifacts-api.elastic.co/v1/search/8.6.0-SNAPSHOT/filebeat?x-elastic-no-kpi=true" version=8.6.0-SNAPSHOT

This happens if no artifacts from the upstream projects have been pushed or the branch is not available in the artifacts-api.

This can be problematic since the merge commits are not moving forward in the CI but failing dramatically, hence the other stages in the CI workflow are not executed.

@mgreau pointed to the artifacts are available in https://artifacts-snapshot.elastic.co/beats/8.6.0-SNAPSHOT/downloads/beats/filebeat/filebeat-8.6.0-SNAPSHOT-linux-x86_64.tar.gz, hence there is no need to use the artifacts-api

Questions

• Should the changes be done to the packaging to use artifacts-snapshot.elastic.co instead of artifacts-api?

cc @narph as you can probably add ore insights

[cmacknz]

I am in favour of fixing CI by switching to artifacts-snapshot.elastic.co unless someone has a strong reason to fix and keep using artifacts-api for this use case. It's not clear what the tradeoffs are, or why we use artifact-api now.

[v1v]

I just removed myself from the assignment as I only raised this issue to let you know what we have found/discussed, but I don't have much knowledge about the elastic agent build system and its internals.

I'd say someone within the Elastic Agent team could be the best candidate to make the required changes in the build system if that's the path forward.

[v1v]

Just to be clear the error happens locally when running mage package, it is unrelated to the CI itself but the build system, the CI is another actor consuming the mage build system

[cmacknz]

Ah I had thought this was a CI issue.

[jlind23]

@v1v I looked at this issue and didn't see any mention of this artifacts api in the elastic-agent repository.
This is the packaging step you mentioned:

elastic-agent/.ci/Jenkinsfile

Line 184 in 13b6e78

steps {

I then looked at the fleet-ci runs and didn't find any failing runs to dig into.

Is this still happening?

[v1v]

@v1v I looked at this issue and didn't see any mention of this artifacts api in the elastic-agent repository.

IIRC, there was a corner case with the existing bumps and the release manager, I don't know if that's still valid

The bulid system in the elastic-agent uses the artifacts-api to grab the versions for some of the beats (metricbeat,filebeat,...), it uses the implementation done in the e2e-testing, see

elastic-agent/magefile.go

Line 911 in 13b6e78

func fetchBinaryFromArtifactsApi(ctx context.Context, packageName, artifact, version, downloadPath string) error {

Hence, if those artifacts are not accessible through the artifacts-api then builds will fail, the release team mentioned to use the artifacts-snapshot.elastic.co instead of artifacts-api

[mgreau]

To help clarify the workflow (yes we need better documentation on this topic - cc @alsayasneh)

• (1) the Artifacts API provides links and metadata for a complete set of snapshot and staging artifacts produced via the unified release workflow (on a ~daily for the snapshot / on schedule for the BCs)
• (2) DRA buckets provide access to the daily (or more) artifacts built asynchronously by each CI workflow. They are available per project from the UI

To understand the differences between these two workflows, let's take a concrete example with the upcoming 8.7 release:

• if you rely on the Artifacts API, you won't have access to 8.7.0 BC1 artifacts until the full BC1 is ready and published. This should happen on Wednesday afternoon if everything goes well because we will need all the teams to have produced their DRA before starting the unified workflow. So if you rely on the 8.7 staging artifacts to build your 8.7 staging artifacts, that won't work using the Artifacts API.
• if you rely on the DRA buckets, then you only need Beats CI workflow to produce 8.7 staging artifacts for building your 8.7 staging artifacts if you have a dependency on this project

[cmacknz]

The Elastic agent main job and local builds that use the EXTERNAL=true option are both currently broken because of this. https://fleet-ci.elastic.co/job/elastic-agent/job/elastic-agent-mbp/job/main/779/console

DEV=true EXTERNAL=true SNAPSHOT=true PLATFORMS=darwin/arm64 PACKAGES=tar.gz mage -v package
Running target: Package
package ran for 56.44111175s
Error: fetchBinaryFromArtifactsApi failed for auditbeat on darwin/arm64: the artifact was not found

This is still affecting us, and is preventing us from having convenient local builds as long as the main snapshot build is failing.

[mgreau]

8.8.0-SNAPSHOT is now available via the Artifacts API https://artifacts-api.elastic.co/v1/versions/8.8.0-SNAPSHOT/builds/8.8.0-5b7507b4

But relying on (2) DRA buckets like here for beats 8.8.0-SNAPSHOT would be better.

[roaksoax]

Synced with @pazone on this, and the current plan of action would be to create a path to use option (2) above, as an alternate path, instead of fully replacing the use of Artifacts API.

Once we have that path available, we can decide whether we can completely drop the use of Artifacts API.

cc @cmacknz any concerns?

[cmacknz]

No concerns from me.

[pazone]

It's almost done but there's a similar pull request: #2727

[pazone]

Now when the snapshot api fails to find a new version we will use artifacts-snapshot.elastic.co as a fallback option. Once this fantastic initiative by @alexsapran is merged the artifacts-snapshot.elastic.co will be used permanently and the dependency on e2e-testing repository will be removed from elastic-agent repo

[cmacknz]

Reopening as this is now blocking all of our builds.

[jlind23]

@cmacknz what happened here? Thought @pazone's fix was working, did anything changed?

[cmacknz]

All agent builds are failing the package step: https://github.com/elastic/elastic-agent/commits/main

The change here was never tested during the post feature freeze period for a minor release, now that we are in that situation there are a few bugs to sort out.

I believe I already have the necessary fixes, will have a set of PRs shortly to close this again.