Mount Go build cache into crossbuild container #9094

swiatekm · 2025-07-21T17:12:18Z

What does this PR do?

Mounts the Go build cache into the golang-crossbuild container. To facilitate this and allow the container to generate the build cache, it also changes the crossbuild container to run as the host user.

This has been tested locally on both Mac and Linux, and the CI passes for this PR. It probably requires somewhat more careful testing, as permission changes in the host build cache could potentially result in breakage for unified releases.

Why is it important?

This reduces the build time of agent during packaging by around 75%. On my machine, the same-architecture build goes from 2 minutes to 25 seconds.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
~~[ ] I have added tests that prove my fix is effective or that my feature works~~
~~[ ] I have added an entry in ./changelog/fragments using the changelog tool~~
~~[ ] I have added an integration test or an E2E test~~

How to test this PR locally

Run mage crossbuild and time it.

Related issues

Relates https://github.com/elastic/observability-dev/issues/4702

dev-tools/mage/build.go

dev-tools/mage/crossbuild.go

v1v

I think we need to test these set of changes in some of the other BK pipelines:

packaging -

elastic-agent/catalog-info.yaml

Lines 268 to 272 in def141d

    
             name: elastic-agent-package 
        
             description: Buildkite pipeline for packaging Elastic Agent package 
        
           spec: 
        
             repository: elastic/elastic-agent 
        
             pipeline_file: ".buildkite/pipeline.elastic-agent-package.yml"

DRA, https://github.com/elastic/elastic-agent/blob/main/catalog-info.yaml#L312-L324

Can you please run them and copy the link to those builds in this PR please?

swiatekm · 2025-07-28T15:27:48Z

@v1v both of them succeeded at the build steps, but failed at the publishing step, even though I set them to dry-run. Not sure if that's in some way an effect of this change or if I made a mistake.

See:

v1v · 2025-07-29T13:23:09Z

That's an expected error, I think it does not support feature branches:

The specified project directory '/release/project-configs/chore/crossbuild-build-cache' does not exist.

https://buildkite.com/elastic/elastic-agent-package/builds/7168#01985144-a389-4509-af44-f1717edd59b2/282-310
https://buildkite.com/elastic/elastic-agent-binary-dra/builds/4211#0198518a-dc7c-4bb0-ad62-f7d65c48a0b9/145-173

As far as I see, you can override them with DRA_BRANCH=main:

elastic-agent/.buildkite/scripts/steps/dra-publish.sh

Line 8 in 8fbdaaa

BRANCH="${DRA_BRANCH:="${BUILDKITE_BRANCH:=""}"}"

I'm gonna run them again:

using the env variables:

DRA_BRANCH="main"
DRA_DRY_RUN="--dry-run"
DRA_PROJECT_ARTIFACT_ID="agent-core"
DRA_PROJECT_ID="elastic-agent-core"
DRA_VERSION=9.2.0
DRA_WORKFLOW="snapshot"

pchila · 2025-07-29T13:50:05Z

both of them succeeded at the build steps, but failed at the publishing step, even though I set them to dry-run. Not sure if that's in some way an effect of this change or if I made a mistake.

You would need to define DRA_BRANCH="main" (or another release branch) and DRA_DRY_RUN="--dry-run" like in this build https://buildkite.com/elastic/elastic-agent-package/builds/7181

Just be careful because without the appropriate DRA_DRY_RUN value you will end up overwriting DRAs built from the release branches with one from your PR

v1v

LGTM. I've just left a comment; however, I'm not an expert in the build system. Please take my +1 as only an acknowledgment.

dev-tools/mage/crossbuild.go

v1v

LGTM. I've just left a comment; however, I'm not an expert in the build system. Please take my +1 as only an acknowledgment.

elasticmachine · 2025-08-05T09:56:39Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

dev-tools/mage/build.go

dev-tools/mage/crossbuild.go

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

dev-tools/mage/crossbuild.go

elastic-sonarqube · 2025-09-11T13:10:55Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

elasticmachine · 2025-09-11T13:34:48Z

💚 Build Succeeded

Buildkite Build
Commit: c9c3e20

History

💛 Build #26553 was flaky 3374450
💛 Build #25819 was flaky 55db0ea
💚 Build #25715 succeeded 61582a5
💛 Build #24545 was flaky b17602c
💔 Build #24544 failed b0fbc2a
💚 Build #24374 succeeded 7b95178

cc @swiatekm

swiatekm · 2025-09-11T14:15:10Z

Looks like the cache is around 5 GB per platform, which is a fair amount.

dev-tools/mage/settings.go

pkoutsovasilis

LGTM, thx for addressing my comments

swiatekm · 2025-09-15T13:46:45Z

I tested the dra publishing pipelines again and didn't see any issues, merging.

github-actions · 2025-09-15T13:47:09Z

@Mergifyio backport 8.18 8.19 9.0 9.1

mergify · 2025-09-15T13:47:38Z

backport 8.18 8.19 9.0 9.1

✅ Backports have been created

#9959 [8.18] (backport #9094) Mount Go build cache into crossbuild container has been created for branch 8.18
#9960 [8.19] (backport #9094) Mount Go build cache into crossbuild container has been created for branch 8.19
#9961 [9.0] (backport #9094) Mount Go build cache into crossbuild container has been created for branch 9.0
#9962 [9.1] (backport #9094) Mount Go build cache into crossbuild container has been created for branch 9.1

* Mount Go build cache into crossbuild container * Run crossbuild container as host user * Update dev-tools/mage/crossbuild.go Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> * Fix formatting * Control mounting mod cache with env variable * Control mounting build cache with env variable * Use a volume for Go build cache --------- Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> (cherry picked from commit fd4de6b)

* Mount Go build cache into crossbuild container * Run crossbuild container as host user * Update dev-tools/mage/crossbuild.go * Fix formatting * Control mounting mod cache with env variable * Control mounting build cache with env variable * Use a volume for Go build cache --------- (cherry picked from commit fd4de6b) Co-authored-by: Mikołaj Świątek <mail@mikolajswiatek.com> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* upstream: (26 commits) fix: ensure EDOT subprocess shuts down gracefully on agent termination (#9886) [main][Automation] Update versions (#9976) Add Collector reference docs and automation (#9953) [beatreceivers] Integrate beatsauthextension (#9257) [main][Automation] Update versions (#9941) Update OTel components to v0.132.0/v1.38.0 (#9954) Enhancement/5235 wrap errors when marking upgrade (#9366) Mount Go build cache into crossbuild container (#9094) Liveness agent state (#9673) [main][Automation] Bump VM Image version to 1757725254 (#9942) Enhancement/5235 correctly wrap errors from copyActionDir and copyRunDirectory (#9349) [main][Automation] Update elastic/beats to afc53c0479ac (#9874) Add -coverpkg option when running unit test to calculate coverage across packages (#9913) Cache binaries downloaded for packaging locally (#9133) [main][Automation] Update versions (#9897) Disable flaky test TestBeatsReceiverLogs (#9891) Allow overriding AGENT_PACKAGE_VERSION and MANIFEST_URL when USE_PACKAGE_VERSION=true (#9864) add ingest-docs team as CODEOWNERS for release notes and docset.yml (#9865) fix: correct spelling of 'output' in various templates and monitoring code (#9827) k8s: Add comment around hostUsers for Universal Profiling deployments (#9847) ...

* Mount Go build cache into crossbuild container * Run crossbuild container as host user * Update dev-tools/mage/crossbuild.go Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> * Fix formatting * Control mounting mod cache with env variable * Control mounting build cache with env variable * Use a volume for Go build cache --------- Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

swiatekm added skip-changelog chore Tasks that just need to be done, they are neither bug, nor enhancements backport-active-all Automated backport with mergify to all the active branches labels Jul 21, 2025

mergify bot assigned swiatekm Jul 21, 2025

swiatekm marked this pull request as ready for review July 28, 2025 13:05

swiatekm requested a review from a team as a code owner July 28, 2025 13:05

swiatekm requested review from a team, pchila, pkoutsovasilis and straistaru July 28, 2025 13:05

swiatekm commented Jul 28, 2025

View reviewed changes

dev-tools/mage/build.go Show resolved Hide resolved

swiatekm commented Jul 28, 2025

View reviewed changes

dev-tools/mage/crossbuild.go Show resolved Hide resolved

v1v reviewed Jul 28, 2025

View reviewed changes

swiatekm requested a review from v1v July 29, 2025 17:32

v1v approved these changes Jul 29, 2025

View reviewed changes

dev-tools/mage/crossbuild.go Outdated Show resolved Hide resolved

v1v previously approved these changes Jul 29, 2025

View reviewed changes

swiatekm dismissed v1v’s stale review via b0fbc2a July 29, 2025 18:28

pierrehilbert added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Aug 5, 2025

swiatekm force-pushed the chore/crossbuild-build-cache branch from b17602c to 61582a5 Compare August 25, 2025 14:35

pkoutsovasilis reviewed Aug 27, 2025

View reviewed changes

dev-tools/mage/build.go Show resolved Hide resolved

dev-tools/mage/crossbuild.go Show resolved Hide resolved

dev-tools/mage/crossbuild.go Outdated Show resolved Hide resolved

swiatekm and others added 5 commits September 10, 2025 14:51

Mount Go build cache into crossbuild container

6704276

Run crossbuild container as host user

315b10b

Update dev-tools/mage/crossbuild.go

2ed0ec6

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

Fix formatting

09d40cc

Control mounting mod cache with env variable

febe8fe

Control mounting build cache with env variable

3374450

swiatekm force-pushed the chore/crossbuild-build-cache branch from 55db0ea to 3374450 Compare September 10, 2025 15:41

pkoutsovasilis reviewed Sep 10, 2025

View reviewed changes

dev-tools/mage/crossbuild.go Outdated Show resolved Hide resolved

Use a volume for Go build cache

c9c3e20

pkoutsovasilis reviewed Sep 11, 2025

View reviewed changes

dev-tools/mage/settings.go Show resolved Hide resolved

pkoutsovasilis approved these changes Sep 11, 2025

View reviewed changes

swiatekm merged commit fd4de6b into main Sep 15, 2025
23 checks passed

swiatekm deleted the chore/crossbuild-build-cache branch September 15, 2025 13:46

This was referenced Sep 15, 2025

[8.18] (backport #9094) Mount Go build cache into crossbuild container #9959

Merged

[8.19] (backport #9094) Mount Go build cache into crossbuild container #9960

Merged

[9.0] (backport #9094) Mount Go build cache into crossbuild container #9961

Merged

mergify bot mentioned this pull request Sep 15, 2025

[9.1] (backport #9094) Mount Go build cache into crossbuild container #9962

Merged

3 tasks

	name: elastic-agent-package
	description: Buildkite pipeline for packaging Elastic Agent package
	spec:
	repository: elastic/elastic-agent
	pipeline_file: ".buildkite/pipeline.elastic-agent-package.yml"

Mount Go build cache into crossbuild container #9094

Mount Go build cache into crossbuild container #9094

Uh oh!

Conversation

swiatekm commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Uh oh!

Uh oh!

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

swiatekm commented Jul 28, 2025

Uh oh!

v1v commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pchila commented Jul 29, 2025

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Aug 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elastic-sonarqube bot commented Sep 11, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Sep 11, 2025

💚 Build Succeeded

History

Uh oh!

swiatekm commented Sep 11, 2025

Uh oh!

Uh oh!

pkoutsovasilis left a comment

Choose a reason for hiding this comment

Uh oh!

swiatekm commented Sep 15, 2025

Uh oh!

Uh oh!

github-actions bot commented Sep 15, 2025

Uh oh!

mergify bot commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

swiatekm commented Jul 21, 2025 •

edited

Loading

v1v commented Jul 29, 2025 •

edited

Loading

mergify bot commented Sep 15, 2025 •

edited

Loading