Fix macOS CI #883

dave-tucker · 2024-02-15T11:28:19Z

This PR contains 3 commits to fix the macOS CI job which has been failing since ~12th February.

The causes was that Rust nightly updated to LLVM v18.1-RC2 but bpf-linker was linked against LLVM 17. Even with the bpf-linker v0.9.11 release (which supports the LLVM RC) macOS CI still fails due to LLVM 18.1-RC2 not being installable via brew to link against. Solving this with homebrew tap/bottles and/or binstalls of bpf-linker is the best long term solution. In the short term we revert to the last working rust nightly toolchain + bpf-linker v0.9.10

This was pretty hard to debug due to missing output from cargo xtask integration-test.
A commit in this PR provides a fix for that too which should make this easier to debug in future.

Note: At some point in CI we were also seeing errors about missing libelf. This seems to be a red herring 🐟
Commits in this PR that added libelf to the brew dependencies have been reverted

netlify · 2024-02-15T11:28:24Z

✅ Deploy Preview for aya-rs-docs ready!

Name	Link
🔨 Latest commit	`6e9dcee`
🔍 Latest deploy log	https://app.netlify.com/sites/aya-rs-docs/deploys/65d770d8a65adb000809c61f
😎 Deploy Preview	https://deploy-preview-883--aya-rs-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

tamird · 2024-02-18T18:23:19Z

Just opened #884 and then saw this. The true failure is coming from bpf-linker but we're not getting its logs (again).

vadorovsky · 2024-02-19T04:40:19Z

@dave-tucker @tamird bpf-linker is fixed, but it uses LLVM 18 now. To fix the latest failure, we need to find some way to install LLVM 18 on macOS.

tamird · 2024-02-19T05:37:37Z

We shouldn't merge this - it doesn't fix anything. We should figure out why we aren't getting errors from bpf-linker.

vadorovsky · 2024-02-19T07:28:40Z

We were getting errors from bpf-linker, because nightly started using LLVM 18, bpf-linker was using LLVM 17, so it ended up trying to link incompatible bitcodes. That was fixed here aya-rs/bpf-linker#184 and in yesterday's 0.9.1 release. And that error is gone after resetting the jobs.

However, the lasest failure is because we don't have LLVM 18 installed on macOS. Homebrew doesn't ship it, which is understandable - it's just a -rc release.

vadorovsky

on hold until we sort out the LLVM 18 on macOS thing

tamird · 2024-02-19T07:33:38Z

The first failing job was https://github.com/aya-rs/aya/actions/runs/7896427886/job/21550389236.

What I'm saying is that I don't see the bitcode error there, it's not clear why the job failed from the logs.

vadorovsky · 2024-02-19T13:44:45Z

The second failing job was https://github.com/aya-rs/aya/actions/runs/7915187581/job/21606490379, not the libelf thing anymore.

error: failed to run custom build command for `integration-test v0.1.0 (/Users/runner/work/aya/aya/test/integration-test)`

Caused by:
  process didn't exit successfully: `/Users/runner/work/aya/aya/target/debug/build/integration-test-014cde848726027a/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at test/integration-test/build.rs:258:9:
  assertion `left == right` failed: cd "/Users/runner/work/aya/aya/test/integration-ebpf" && env -u RUSTC -u RUSTUP_TOOLCHAIN CARGO_CFG_BPF_TARGET_ARCH="x86_64" "cargo" "build" "-Z" "build-std=core" "--bins" "--message-format=json" "--release" "--target" "bpfel-unknown-none" "--target-dir" "/Users/runner/work/aya/aya/target/x86_64-unknown-linux-musl/debug/build/integration-test-bc58de130b9ea635/out/integration-ebpf" failed: ExitStatus(unix_wait_status(25856))
    left: Some(101)
   right: Some(0)
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: AYA_BUILD_INTEGRATION_BPF="true" "cargo" "build" "--message-format=json" "--target" "x86_64-unknown-linux-musl" "--config" "target.x86_64-unknown-linux-musl.linker = \"rust-lld\"" "--package" "integration-test" "--tests" "--profile" "dev" failed: ExitStatus(unix_wait_status(25856))
Error: Process completed with exit code 123.

Our bad for not logging the whole stderr/out, but this is for sure the bitcode linking mismatch we've seen in other places.

tamird · 2024-02-19T13:51:28Z

Yes, this is exactly what I'm saying: we can't see the actual error.

dave-tucker · 2024-02-19T14:02:35Z

@tamird see the last commit I just pushed. It seems cargo::warning decided not print anything after a newline character...
So we have to manually print the longer message line-by-line to get the error output from build.rs 😓

With the change applied a failing run now produces the following output, which includes the error message that @vadorovsky provided:

arning: integration-test@0.1.0:   = note: 14:01:43 [ERROR] llvm: Invalid record (Producer: 'LLVM18.1.0-rust-1.78.0-nightly' Reader: 'LLVM 17.0.6')
warning: integration-test@0.1.0:           error: failure linking module core-96a21db970ed0e7f.core.1934461c7d58be1e-cgu.00.rcgu.o from /Users/dave/dev/aya/target/aarch64-unknown-linux-musl/debug/build/integration-test-0da7decb9fffdd68/out/integration-ebpf/bpfel-unknown-none/release/deps/libcore-96a21db970ed0e7f.rlib14:01:43 [WARN] ignoring file "/var/folders/yg/xdvg2sbs6fj19g06_ttyz1n40000gn/T/rustcVLTxyq/symbols.o": no embedded bitcode
warning: integration-test@0.1.0:           14:01:43 [WARN] ignoring archive item "lib.rmeta": no embedded bitcode
warning: integration-test@0.1.0:           14:01:43 [WARN] ignoring archive item "lib.rmeta": no embedded bitcode
warning: integration-test@0.1.0:           14:01:43 [WARN] ignoring archive item "lib.rmeta": no embedded bitcode
warning: integration-test@0.1.0:           14:01:43 [WARN] ignoring archive item "lib.rmeta": no embedded bitcode
warning: integration-test@0.1.0:           14:01:43 [WARN] ignoring archive item "lib.rmeta": no embedded bitcode
warning: integration-test@0.1.0:           14:01:43 [WARN] ignoring archive item "lib.rmeta": no embedded bitcode
warning: integration-test@0.1.0:           14:01:43 [WARN] ignoring archive item "lib.rmeta": no embedded bitcode
warning: integration-test@0.1.0:           14:01:43 [WARN] ignoring archive item "lib.rmeta": no embedded bitcode
warning: integration-test@0.1.0:           
warning: integration-test@0.1.0: 
warning: integration-test@0.1.0: 
warning: integration-test@0.1.0: error: aborting due to 1 previous error
warning: integration-test@0.1.0: 
warning: integration-test@0.1.0: 
warning: integration-test@0.1.0: error: could not compile `integration-ebpf` (bin "log") due to 2 previous errors

error: failed to run custom build command for `integration-test v0.1.0 (/Users/dave/dev/aya/test/integration-test)`

Caused by:
  process didn't exit successfully: `/Users/dave/dev/aya/target/debug/build/integration-test-f879e55cb8bd8929/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at test/integration-test/build.rs:260:9:
  assertion `left == right` failed: cd "/Users/dave/dev/aya/test/integration-ebpf" && env -u RUSTC -u RUSTUP_TOOLCHAIN CARGO_CFG_BPF_TARGET_ARCH="aarch64" "cargo" "build" "-Z" "build-std=core" "--bins" "--message-format=json" "--release" "--target" "bpfel-unknown-none" "--target-dir" "/Users/dave/dev/aya/target/aarch64-unknown-linux-musl/debug/build/integration-test-0da7decb9fffdd68/out/integration-ebpf" failed: ExitStatus(unix_wait_status(25856))
    left: Some(101)
   right: Some(0)
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: AYA_BUILD_INTEGRATION_BPF="true" "cargo" "build" "--message-format=json" "--target" "aarch64-unknown-linux-musl" "--config" "target.aarch64-unknown-linux-musl.linker = \"rust-lld\"" "--package" "integration-test" "--tests" "--profile" "dev" failed: ExitStatus(unix_wait_status(25856))

tamird · 2024-02-19T14:04:36Z

Nasty! We have the same code in xtask/src/run.rs that we should also fix.

tamird · 2024-02-19T16:04:15Z

.github/workflows/ci.yml

@@ -234,7 +234,7 @@ jobs:
          find /usr/local/bin -type l -exec sh -c 'readlink -f "$1" \
          | grep -q ^/Library/Frameworks/Python.framework/Versions/' _ {} \; -exec rm -v {} \;
          HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=1 \
-          brew install dpkg findutils gnu-tar llvm pkg-config qemu
+          brew install dpkg findutils gnu-tar llvm pkg-config libelf qemu


we should remove this commit?

afaict we still need it - it was an implicit dependency of libbpf before but we didn't install it.
once macOS CI is green again (which i'm hoping my last commit should do) i'll do an experiment to see whether it is still required.

please drop a note here when you learn the answer

Well, it seems I was wrong 😢 libelf isn't needed.
Commit reverted. I'm unable to find or reproduce the error that led me to believe it was libelf in the first place.

It logs a warning :)

tamird · 2024-02-22T12:54:17Z

(reminder to update the PR title plz)

dave-tucker · 2024-02-22T14:32:49Z

Note: Clippy will fail until #885 has merged. We can merge either first, but one will be red (unless I incorporate the clippy fix here too).

.github/workflows/ci.yml

The cargo::warning seems to ignore output after a newline. Iterate over the entire rendered message and print it line-by-line. Signed-off-by: Dave Tucker <dave@dtucker.co.uk>

LLVM-18 hasn't been released on macOS yet Signed-off-by: Dave Tucker <dave@dtucker.co.uk>

dave-tucker · 2024-02-22T16:57:18Z

Merging to get CI back to green!

mergify bot added the test A PR that improves test cases or CI label Feb 15, 2024

tamird mentioned this pull request Feb 18, 2024

Install libelf on macOS #884

Closed

vadorovsky approved these changes Feb 19, 2024

View reviewed changes

vadorovsky requested changes Feb 19, 2024

View reviewed changes

dave-tucker force-pushed the libelf branch from 53fcdc0 to 5380a95 Compare February 19, 2024 14:32

tamird approved these changes Feb 19, 2024

View reviewed changes

dave-tucker force-pushed the libelf branch 2 times, most recently from e9cd636 to 322c883 Compare February 22, 2024 12:51

dave-tucker force-pushed the libelf branch from 322c883 to 579660b Compare February 22, 2024 14:25

dave-tucker changed the title ~~ci: Install libelf on macOS~~ Fix macOS CI Feb 22, 2024

tamird reviewed Feb 22, 2024

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

dave-tucker force-pushed the libelf branch from 579660b to 8c41658 Compare February 22, 2024 14:43

dave-tucker changed the title ~~Fix macOS CI~~ DO NOT MERGE (YET): Fix macOS CI Feb 22, 2024

dave-tucker marked this pull request as draft February 22, 2024 15:56

dave-tucker added 2 commits February 22, 2024 16:05

integration-test: Fix build.rs output

0c58bb6

The cargo::warning seems to ignore output after a newline. Iterate over the entire rendered message and print it line-by-line. Signed-off-by: Dave Tucker <dave@dtucker.co.uk>

ci: Use nightly-2022-02-12 on macOS

6e9dcee

LLVM-18 hasn't been released on macOS yet Signed-off-by: Dave Tucker <dave@dtucker.co.uk>

dave-tucker force-pushed the libelf branch from afa402d to 6e9dcee Compare February 22, 2024 16:05

dave-tucker changed the title ~~DO NOT MERGE (YET): Fix macOS CI~~ Fix macOS CI Feb 22, 2024

dave-tucker marked this pull request as ready for review February 22, 2024 16:56

dave-tucker merged commit 664bb47 into aya-rs:main Feb 22, 2024
21 checks passed

dave-tucker deleted the libelf branch February 22, 2024 16:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix macOS CI #883

Fix macOS CI #883

dave-tucker commented Feb 15, 2024 •

edited

Loading

netlify bot commented Feb 15, 2024 •

edited

Loading

tamird commented Feb 18, 2024

vadorovsky commented Feb 19, 2024

tamird commented Feb 19, 2024

vadorovsky commented Feb 19, 2024

vadorovsky left a comment

tamird commented Feb 19, 2024

vadorovsky commented Feb 19, 2024

tamird commented Feb 19, 2024

dave-tucker commented Feb 19, 2024

tamird commented Feb 19, 2024

tamird Feb 19, 2024

dave-tucker Feb 22, 2024

tamird Feb 22, 2024

dave-tucker Feb 22, 2024

tamird Feb 22, 2024

tamird commented Feb 22, 2024

dave-tucker commented Feb 22, 2024

dave-tucker commented Feb 22, 2024

Fix macOS CI #883

Fix macOS CI #883

Conversation

dave-tucker commented Feb 15, 2024 • edited Loading

netlify bot commented Feb 15, 2024 • edited Loading

✅ Deploy Preview for aya-rs-docs ready!

tamird commented Feb 18, 2024

vadorovsky commented Feb 19, 2024

tamird commented Feb 19, 2024

vadorovsky commented Feb 19, 2024

vadorovsky left a comment

Choose a reason for hiding this comment

tamird commented Feb 19, 2024

vadorovsky commented Feb 19, 2024

tamird commented Feb 19, 2024

dave-tucker commented Feb 19, 2024

tamird commented Feb 19, 2024

tamird Feb 19, 2024

Choose a reason for hiding this comment

dave-tucker Feb 22, 2024

Choose a reason for hiding this comment

tamird Feb 22, 2024

Choose a reason for hiding this comment

dave-tucker Feb 22, 2024

Choose a reason for hiding this comment

tamird Feb 22, 2024

Choose a reason for hiding this comment

tamird commented Feb 22, 2024

dave-tucker commented Feb 22, 2024

dave-tucker commented Feb 22, 2024

dave-tucker commented Feb 15, 2024 •

edited

Loading

netlify bot commented Feb 15, 2024 •

edited

Loading