Download build artifacts from the backport branch for testing in the main branch

[leofang]

Close #329.

Update: Please see #357 (comment).

Refresher: There are two kinds of caches that we can use in GHA, Cache and Artifacts. We've been using Artifacts to store build artifacts, which works fine so far but the main issue is the artifacts are scoped on a per-PR basis, meaning they cannot be reused across CI workflow runs triggered by different PRs.

This PR adds the capability of uploading artifacts to the Cache space when a PR is merged into the main branch, so that they can serve as a fallback when a workflow needs certain artifacts for whatever reason. Note that while the Cache space is limited to 10 GB per repo, for our purpose (we have small wheels) it is still OK as a stop-gap solution, until our DevOp team finds a more sustainable one.

I also cleaned up the shell choice a bit so that all job steps use the same setting.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[leofang]

/ok to test

[leofang]

/ok to test

[leofang]

/ok to test

[leofang]

/ok to test

[leofang]

/ok to test

[leofang]

/ok to test

[leofang]

/ok to test

[leofang]

/ok to test

[leofang]

/ok to test

[leofang]

/ok to test

[leofang]

/ok to test

[jakirkham]

Note that while the Cache space is limited to 10 GB per repo, for our purpose (we have small wheels) it is still OK as a stop-gap solution, until our DevOp team finds a more sustainable one.

Think there is a way to setup a retention policy in repo settings or the workflow. So this might be an option to help manage size

Should add without this setting artifacts stick around forever. Removing them in the UI is one at a time. So something to be aware of

[leofang]

/ok to test

[leofang]

Thanks, @jakirkham! None of GHA Cache related comments are relevant anymore since I am moving away from it (#357 (comment)). I'll update the PR title/description once the CI is green.

[leofang]

/ok to test

[leofang]

OK commit 75e37bd is basically a rewrite of the whole PR. The result is indeed a lot cleaner as expected. Since we have the capability of fetching artifacts generated from the backport branch (11.8.x), there is no need to introduce any concept such as a global cache as discussed earlier, and therefore we can avoid using the GHA Cache which adds some complexity.

The new logic is simply:

• For testing against CTK 12.x:
  • Run tests of cuda.core and cuda.bindings against cuda.bindings built from the main branch
• For testing against CTK 11.x:
  • Run tests of cuda.core against the most recent successful build of cuda.bindings from the 11.8.x branch
  • tests of cuda.bindings are run in the 11.8.x branch when we backport a PR

@ksimpson-work @vzhurba01 this is ready for review.

[leofang]

Note: this hack can be removed now that we generate Python 3.13 wheels for cuda.bindings 11.8 and can retrieve them in the CI; we do not need them published on PyPI in order to use them!

[ksimpson-work]

Correct me if I'm wrong. I want to understand this well.

I see two cases:

One where you are backporting something to 11.8.x where you would want to test cuda core against the active 11.8.x CI build, in which case you would want to target the latest 11.8.x if it was successfully built, or bail out if there were build errors.

Second case is making a change in main, specifically to cuda.core, in which case you would want to test against, not the latest successful CI on 11.8.x, but the top of the 11.8.x tree. This is because if someone was simultaneously testing an 11.8.x change, you might test against an 11.8.x version that is different from what a user would be installing.

From my understanding of this change, there's a race condition between (11.8.x CI workflows + merges) and main workflows.

WDYT?

[leofang]

Race condition is a legit concern but it is still better than the status quo (no integration test against the head of the backport branch). Moreover, we will set up a nightly CI to reduce the risk (#294) and we already have pre-release QA as the final defense line, so I think it is not very risky and can be improved once our DevOp team take over and iterate toward a more robust implementation.

In the first case, if a backport is relevant for cuda.core to work, cuda.core tests would fail unless the backport is merged and rebuilt. So we will know what's going on without a silent green light.

The second case is where the race condition could happen IIUC ("which 11.8 build am I testing against?").

[ksimpson-work]

Ok, I understand that it is a catch 22, and agree that testing against the latest success is far more robust than not testing at all. I just wanted to verbalize that to ensure I correctly understood + make sure we understood that there is a possible improvement there for the DevOps team to address in the future. LGTM

[leofang]

to ensure I correctly understood + make sure we understood that there is a possible improvement there for the DevOps team to address in the future.

Yes all great questions here! You made me think twice (and long enough to seek for an alternative solution). This is why we need code review 😄 Thanks, Keenan!