kcov fails ("status: 0x85") reproducibly on some binaries (tracker for kcov bug 339)

[ggould-tri]

In effect this issue tracks the fact that all of our supported platforms use versions of kcov that suffer from bug SimonKagstrom/kcov#339 . This bug causes binaries with a particular set of bytes at the beginning to be incorrectly parsed as shell scripts, causing them to fail in overnight CI; it is silenced by making any small change to the code that causes the binary bytes to change -- reordering the individual tests in a test suite, for instance.

If this issue occurs, the log will contain the string:

kcov: warning: Other status: 0x85

This issue is tagged as "component: continuous integration" because it is almost invariably discovered by the buildcop the next morning and so should be on the buildcop's search list of common failures.

Examples of this problem in the past (there are many more occurrences but most are resolved over slack and don't result in issues being filed):

• Need a workaround of a rare kcov bug that showed up in CI Nightly (kcov: warning: Other status: 0x85). #14824
• Spurious focal kcov failure on //math:gray_code_test #14424

[jwnimmer-tri]

... reordering the individual tests in a test suite, for instance.

Changing the order of tests ends up being a larger-than-necessary patch, disrupting history.

My suggestion for the least intrusive work-around is to add and remove namespaces in the test program (possibly inline namespaces) until it works (see, e.g., #17724, #17835).

[jwnimmer-tri]

Both Ubuntu and Debian seem to have stalled out w.r.t. kcov and are no longer packaging new versions.

In case we decide we need a version upgrade, we might consider using https://github.com/bazelbuild/rules_foreign_cc to build kcov from source, instead of relying on Ubuntu. As long as the (single-threaded) build doesn't take inordinately long, there's a fair chance it would just work out of the box.

[rpoyner-tri]

On a vintage puget, the make -j 40 step took under 10 seconds to compile and link from scratch. The cmake .. step is maybe 2 seconds. So, total from-scratch build budget should be small if configured right.

[jwnimmer-tri]

At the moment rules_foreign_cc maxes out at -j 1 (bazel-contrib/rules_foreign_cc#329). There's finally been a bit of progress on doing better (bazelbuild/bazel#10443) but for the moment the metric we care about is the make -j 1 latency vs how patient people are willing to be, and how well Bazel does at not spuriously invalidating the cached build of kcov.

[rpoyner-tri]

The -j 1 compile and link was ~70 seconds on vintage puget, so, maybe not great if it triggers often.

[jwnimmer-tri]

Closing as duplicate of #18969. We can still cite this issue in fixup PRs, but there's no action to take here.