[CI] Speed up wheel commit validation check by 100x

[can-anyscale]

Why are these changes needed?

Speed up wheel commit validation check by 100x. Also hopefully will alleviate if not eliminate the 'Observed wheel commit () is not expected' issue (#32156) that has been creeping through many of ci/cd builds in our pipeline.

The existing code uses pipe to read from a rather large file (>50MB). Pipe however has buffer limit which by default in term of kb (https://man7.org/linux/man-pages/man7/pipe.7.html) so what we look for might not exist. We can fix this by tell unzip the exact file we are looking for. That file is pretty small so we should not hit buffer limit.

You might notice other surpises might still happen with this fix (e.g. many files that match ^commit). This sanity check goes back to 2 years ago by our veteran Kai (234b015) to sanity check issues with stale artifacts from previous builds or race conditions between builds. Further investigation on how builkite agent multi-tenant is setup might or might not simplify this logic further.

Signed-off-by: Cuong Nguyen can@anyscale.com
@can-anyscale

Related issue number

#32156

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• [] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • CI Tests

[aslonnie]

linux pipe is a streaming setup, right? why the buffer size matters?

[aslonnie]

are you able to reproduce the issue and find out the root cause?

[can-anyscale]

@aslonnie good point, and you're absolutely correct, the stdin is stream; purely a theory but my best guess is streaming such a large stdin makes buffer and grep unpredictable, and hopefully no harm to reduce the data usage a bit

I can find and download the problematic wheel (e.g. see artifacts and download cp39 in https://buildkite.com/ray-project/oss-ci-build-branch/builds/2791#0186ee52-dbab-4169-839c-f8eab80791be), but running the same grep command locally returns a proper commit, so the wheel is ok, but the empty commit return is not reproducible

unzip -p ray-3.0.0.dev0-cp39-cp39-manylinux2014_aarch64.whl | grep "^commit" | awk '-F"' '{print $2}'
03a9d21

unzip -p ray-3.0.0.dev0-cp39-cp39-manylinux2014_aarch64.whl "*/ray/init.py" | grep "^commit" | awk '-F"' '{print $2}'
03a9d21

The second command is also much (100x) faster

[cadedaniel]

I'm sus of the hypothesis that large files would cause grep to fail. But the speed up is great 😀 maybe change PR title to something like "speed up commit extraction from wheel"?

[cadedaniel]

Can we add a log to print which wheel file it came from?

[can-anyscale]

Absolutely!

[cadedaniel]

cc @jjyao

[can-anyscale]

w00h00 thank you both!