Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

try a new approach to reuse cached builds #67

Draft
wants to merge 3 commits into
base: develop
Choose a base branch
from

Conversation

bcressey
Copy link
Contributor

@bcressey bcressey commented Aug 6, 2024

Issue number:
Follow-up on #65

Description of changes:
Try a new strategy to force use of cache:

  1. Rebuild everything with a new buildsys flag that skips actual builds
  2. Obtain a list of modified files
  3. Touch all modified files so the corresponding crates will rebuild
  4. Launch a build without the special buildsys flag

Testing done:
😬

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@bcressey bcressey force-pushed the cache-resync branch 2 times, most recently from 32bd1c2 to 8ab81c4 Compare August 6, 2024 23:20
@bcressey
Copy link
Contributor Author

bcressey commented Aug 7, 2024

With the above algorithm and a modified systemd package, the expected rebuilds are:

   Compiling systemd v0.2.0 (/home/fedora/corekit/packages/systemd)
   Compiling login v0.0.1 (/home/fedora/corekit/packages/login)
   Compiling wicked v0.1.0 (/home/fedora/corekit/packages/wicked)
   Compiling mdadm v0.1.0 (/home/fedora/corekit/packages/mdadm)
   Compiling dbus-broker v0.1.0 (/home/fedora/corekit/packages/dbus-broker)
   Compiling docker-engine v0.1.0 (/home/fedora/corekit/packages/docker-engine)
   Compiling netdog v0.1.0 (/home/fedora/corekit/packages/netdog)
   Compiling release v0.0.0 (/home/fedora/corekit/packages/release)
   Compiling bottlerocket-core-kit v0.1.0 (/home/fedora/corekit/kits/bottlerocket-core-kit)

These packages all use the bundled Go modules feature, which may be causing spurious rebuilds:

   Compiling aws-iam-authenticator v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/aws-iam-authenticator)
   Compiling aws-signing-helper v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/aws-signing-helper)
   Compiling cni v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/cni)
   Compiling ecr-credential-provider v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/ecr-credential-provider)
   Compiling ecr-credential-provider-1_27 v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/ecr-credential-provider-1.27)
   Compiling ecr-credential-provider-1_29 v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/ecr-credential-provider-1.29)
   Compiling ecr-credential-provider-1_30 v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/ecr-credential-provider-1.30)
   Compiling oci-add-hooks v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/oci-add-hooks)
   Compiling soci-snapshotter v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/soci-snapshotter)

These package rebuilds are a mystery, though at least it's consistent for both x86_64 and aarch64 runners:

   Compiling kubernetes-1_23 v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/kubernetes-1.23)
   Compiling kubernetes-1_24 v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/kubernetes-1.24)
   Compiling kubernetes-1_25 v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/kubernetes-1.25)
   Compiling kubernetes-1_26 v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/kubernetes-1.26)
   Compiling kubernetes-1_27 v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/kubernetes-1.27)
   Compiling kubernetes-1_28 v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/kubernetes-1.28)
   Compiling kubernetes-1_29 v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/kubernetes-1.29)
   Compiling kubernetes-1_30 v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/kubernetes-1.30)
   Compiling kmod-5_10-nvidia v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/kmod-5.10-nvidia)
   Compiling kmod-5_15-nvidia v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/kmod-5.15-nvidia)
   Compiling kmod-6_1-nvidia v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/kmod-6.1-nvidia)

@bcressey bcressey force-pushed the cache-resync branch 4 times, most recently from c5b1630 to be4a777 Compare August 13, 2024 20:32
Signed-off-by: Ben Cressey <bcressey@amazon.com>
Signed-off-by: Ben Cressey <bcressey@amazon.com>
Signed-off-by: Ben Cressey <bcressey@amazon.com>
@bcressey
Copy link
Contributor Author

Trying a run with a couple twoliter changes:

  1. BUILDSYS_CICD_HACK=1 to skip over actual builds in the initial pass
  2. bcressey/twoliter@4c4e9ad to set modification time for external files, to avoid the spurious rebuilds

@bcressey
Copy link
Contributor Author

OK, with the twoliter changes I see this behavior:

   Compiling systemd v0.2.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/systemd)
   Compiling login v0.0.1 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/login)
   Compiling wicked v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/wicked)
   Compiling dbus-broker v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/dbus-broker)
   Compiling mdadm v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/mdadm)
   Compiling docker-engine v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/docker-engine)
   Compiling netdog v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/netdog)
   Compiling release v0.0.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/packages/release)
   Compiling bottlerocket-core-kit v0.1.0 (/home/runner/work/bottlerocket-core-kit/bottlerocket-core-kit/kits/bottlerocket-core-kit)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 38s

... just the set of packages we expect to be rebuilt! 🎉

@bcressey
Copy link
Contributor Author

We're now on a version of Twoliter that supports this hack, so it's getting closer.

I realized a shortcoming of the approach here: it only handles file-level modifications that would trigger rebuilds. The environment will be the same for both builds, so we won't do rebuilds for any of the cargo:rerun-if-env-changed variables.

The full set:

cargo:rerun-if-env-changed=BUILDSYS_ARCH
cargo:rerun-if-env-changed=BUILDSYS_EXTERNAL_KITS_DIR
cargo:rerun-if-env-changed=BUILDSYS_OUTPUT_GENERATION_ID
cargo:rerun-if-env-changed=BUILDSYS_PACKAGES_DIR
cargo:rerun-if-env-changed=BUILDSYS_ROOT_DIR
cargo:rerun-if-env-changed=BUILDSYS_STATE_DIR
cargo:rerun-if-env-changed=TLPRIVATE_SDK_IMAGE

Of these, the most interesting is TLPRIVATE_SDK_IMAGE. The others are unlikely to change, but updating the SDK image needs to drive a full rebuild.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant