Biannual linux kernel/security update for CI image

[BetsyMcPhail]

Previous issue: #18617

[BetsyMcPhail]

All Jammy and Noble images have been updated. Both provisioned images were also updated and re-provisioned.

[BetsyMcPhail]

Reopening this as something went wrong with the Jammy update, see https://drakedevelopers.slack.com/archives/C270MN28G/p1720809110296759

[BetsyMcPhail]

An example of the failure with the initial update is (from https://drake-jenkins.csail.mit.edu/view/Pull%20Request/job/linux-jammy-clang-bazel-experimental-leak-sanitizer/4202/consoleFull)

[12:40:14 PM]  ==================== Test output for //common:text_logging_no_spdlog_test:
[12:40:14 PM]  /media/ephemeral0/ubuntu/workspace/linux-jammy-clang-bazel-experimental-leak-sanitizer/_bazel_ubuntu/fefafcca0afccc826f494811f9e41d82/sandbox/linux-sandbox/918/execroot/drake/bazel-out/k8-dbg/bin/common/text_logging_no_spdlog_test.runfiles/drake/tools/dynamic_analysis/lsan: line 7:    20 Segmentation fault      (core dumped) "$@"
[12:40:14 PM]  ================================================================================
[12:40:14 PM]  FAIL: //common:text_logging_no_spdlog_test (see /media/ephemeral0/ubuntu/workspace/linux-jammy-clang-bazel-experimental-leak-sanitizer/_bazel_ubuntu/fefafcca0afccc826f494811f9e41d82/execroot/drake/bazel-out/k8-dbg/testlogs/common/text_logging_no_spdlog_test/test.log)

[BetsyMcPhail]

I tried the update again, spun up an instance from the updated image, ran the tests and saw the same results.

Note this error only appears after updating the kernel. Updating the unprovisioned image (apt update and upgrade, ignoring the held back linux-aws packages) and then rebuilding the provisioned image (running setup scripts) seems to be fine.

On my local machine, I also get the error when running: test --config=clang --compilation_mode=dbg --config=lsan //common/...

@jwnimmer-tri when you have a chance, can you see if the above command succeeds for you locally?

[jwnimmer-tri]

Locally, I'm currently using this kernel:

jwnimmer@call-cps:~$ uname -a
Linux call-cps 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Running bazel test --config=clang --compilation_mode=dbg --config=lsan //common/... with the current master branch crashes for me with those same kind of segfaults.

Drilling down and running bazel test --config=clang --compilation_mode=dbg --config=lsan //common:fmt_test --runs_per_test=100 --nocache_test_results, I get FAILED in 24 out of 100 in 0.7s.

I'll see if I can figure out where things go wrong.

[jwnimmer-tri]

I wasn't able to get any traction. The corefiles don't work, and LSan programs apparently can't be run under gdb. Even the segfault doesn't appear to have any details that I can google around for.

I think you might have to set up an experiment somewhere where you bisect through different kernel versions, and see if that sheds any light.

[BetsyMcPhail]

Interesting... I will try to narrow down which version it started failing.