Run tests against nightly CLI bundles

[edoardopirovano]

This PR changes the integration tests of the CodeQL Action to also run against a recent nightly build of the CodeQL bundle. This should help catch compatibility issues arising from changes in the Action earlier.

[adityasharad]

I'd like to still be testing against latest because it exercises the hardcoded default configured within the Action repo, and sometimes may be different from both null and nightly. Can we matrix over all three?

[edoardopirovano]

Yep, done!

[adityasharad]

Looks good, thanks! Hopefully won't make too many matrixed jobs, otherwise we might have to split up the workflow. Merge after the nightly bundle is published.

[adityasharad]

Minor: You may want to pull this one into an env var too.

[edoardopirovano]

Thanks, done.

[edoardopirovano]

There's actually already a nightly release up from the workflow's testing, but the PR check is failing because https://github.com/github/semmle-code/pull/39627 isn't merged yet I think. We'll have to wait for the next nightly build after that's merged before this can go in 🙂

[aeisenberg]

I'm a little bit confused. This looks like it only runs the init action with the nightly CLI. Should we be running analyze as well?

[edoardopirovano]

I'm a little bit confused. This looks like it only runs the init action with the nightly CLI. Should we be running analyze as well?

The way these tests are setup is a little weird. We use the init action because it gets the CodeQL CLI and puts it on disk so we can print out the versions that we are going to be running with in the later step. Then, these all get put in the VERSIONS_JSON matrix and all the later tests are run with all the versions in there. I'm not a huge fan of this structure, but it didn't seem necessary to change it.