CI run-tests script now attempts to detect Apple Clang 15 and other unsupported regression-tests compiler versions

[bluetarpmedia]

After investigating some spurious macOS regression-test job failures, I found that the macos-13 GitHub runner has both Apple Clang 14 and 15 installed.

It appears that on some runs, running clang++ incorrectly selects Apple Clang 15, which is currently unsupported as a compiler version for the regression-tests. For example, see this log file for this failed job.

This PR makes 2 changes:

• If Apple Clang 15 is detected it tries to switch to 14 instead
• Prints an error message and aborts the job if an unsupported compiler version is found

CC @jarzec

[gregmarr]

Is there a reason to not add support for Clang 15?

[hsutter]

incorrectly selects Apple Clang 15, which is currently unsupported as a compiler version for the regression-tests

Is there an issue with Apple Clang 15, or just that there's no subdirectory for Apple Clang 15 in the test results?

[jarzec]

@hsutter In fact I wanted to propose adding support for Clang 15. I think the run-test.sh script has some leftovers of my trials for that. This version is available on Ubuntu 2204 and is installed by default on the GitHub runner. This is also the case on the macos-13 runner. The added value of adding this version is that all regression tests can be compiled and run using it, so it would be the only compiler on POSIX systems allowing for that (at least that was the case last time I tried). Can quickly prepare a PR with that.

[jarzec]

[Question] Are you sure Apple Clang 15 was selected? The GitHub runner states that Clang 15 s installed from Homebrew. I checked and the output of $(brew --prefix llvm@15)/bin/clang --version is:

Homebrew clang version 15.0.7
Target: x86_64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /usr/local/opt/llvm@15/bin

[bluetarpmedia]

No, I'm not 100% sure, this is very much some investigative work!

All I can say for certain right now is that on some macOS job runs (e.g. the one linked above, and I've seen it other times too), clang++ does not resolve to Apple Clang 14. That's why the script produces the not found for compiler: 'clang++' error message.

One of the updates in this PR is to provide more info about why the job fails. If this PR is merged then in the future we'll get more info due to this:

printf "Unhandled compiler version:\n$compiler_version\n\n"

That may show Apple clang 15.0 etc or something else.

Yeah, the macos-13 runner confusingly has both LLVM Clang installed as well as multiple versions of Xcode (with Apple Clang).

Xcode 14.3 is meant to be the default so clang++ would resolve to Apple Clang 14, but I suspect there's a problem which is causing Xcode 15 to be active. But, it could also somehow be invoking LLVM Clang 15. To me that seemed less likely because, like you noted, it's installed via Homebrew and needs the brew --prefix llvm@15 to get the correct path.

[bluetarpmedia]

Is there a reason to not add support for Clang 15?

Nope, that's definitely an option. (Note there's Clang 15 as well as Apple Clang 15.)

[bluetarpmedia]

Is there an issue with Apple Clang 15, or just that there's no subdirectory for Apple Clang 15 in the test results?

Yeah, the latter, because there are no expected test-results yet for Apple Clang 15.

Even if we add test-results for Clang 15 and Apple Clang 15, I think it's still worth adding this part of the PR:

printf "Unhandled compiler version:\n$compiler_version\n\n"

That way in the future we'll get more details when the job fails due to a new compiler on the GitHub runner that we don't yet have test-results for.

[hsutter]

@jarzec I see an "approved these changes" from you midway up above, just checking: Does this look ready to merge to you?

[jarzec]

@jarzec I see an "approved these changes" from you midway up above, just checking: Does this look ready to merge to you?

Yes, please do merge. I've started to look into including Clang 15 on top of the updates here.
(Sorry, I previously responded from my work account).

[hsutter]

Thanks!