Fixincludes problem with SDKs 16.0

[simonjwright]

XCode & CLT 16.0 were released with macOS 14.7, and now we get e.g.

$ gcc hello.c
In file included from hello.c:1:
/opt/gcc-14.2.0-1-aarch64/lib/gcc/aarch64-apple-darwin21/14.2.0/include-fixed/stdio.h:83:8: error: unknown type name ‘FILE’
   83 | extern FILE *__stdinp;
      |        ^~~~
/opt/gcc-14.2.0-1-aarch64/lib/gcc/aarch64-apple-darwin21/14.2.0/include-fixed/stdio.h:81:1: note: ‘FILE’ is defined in header ‘<stdio.h>’; this is probably fixable by adding ‘#include <stdio.h>’
   80 | #include <sys/_types/_seek_set.h>
  +++ |+#include <stdio.h>
   81 | 
/opt/gcc-14.2.0-1-aarch64/lib/gcc/aarch64-apple-darwin21/14.2.0/include-fixed/stdio.h:84:8: error: unknown type name ‘FILE’
   84 | extern FILE *__stdoutp;
      |        ^~~~
.....

What would help?

[iains]

there are some fixes queued up - waiting for some time to do release tests - I have, however, pushed the ones under test to the gcc-14-2-darwin branch (but not yet made a release) - is that what you've tested?

[iains]

@fxcoudert - ISTR you had a fix for this ? (I guess that also needs back-porting)

[iains]

@fxcoudert - ISTR you had a fix for this ? (I guess that also needs back-porting)

I think it is r15-1697-g1dc14318155057 - will look at doing the back port to 14. over the weekend

[simonjwright]

there are some fixes queued up - waiting for some time to do release tests - I have, however, pushed the ones under test to the gcc-14-2-darwin branch (but not yet made a release) - is that what you've tested?

No, mine was gcc-14.2-darwin-r1

[edit] I don’t recognise the "gcc-14.2-darwin-r1 branch"?

[edit2] I meant to say "gcc-14.2-darwin branch" -- which I've now found, after a 'git fetch'

[fxcoudert]

XCode & CLT 16.0 were released with macOS 14.7, and now we get

Is that error from a fresh compilation against the new Xcode and CLT? Or is that using a compiler built against CLT 15 on a system with CLT 16? Also, what SDK version was it compiled against, and is it being run against?

I have one fix related to stdio.h currently, which is waiting further testing from @iains before it can be pushed, and it is this: https://gist.github.com/fxcoudert/28502eaad49b6a8996447387df99733a

[simonjwright]

Is that error from a fresh compilation against the new Xcode and CLT? Or is that using a compiler built against CLT 15 on a system with CLT 16? Also, what SDK version was it compiled against, and is it being run against?

The compiler was built against CLT 15.3 and was reported to me having being run on a system with macOS 14.7 and Xcode 16.0. I find the same behaviour with macOS 15.0.1 and CLT 16.0 (and also with GCC 13.2, built from gcc-13.2-darwin-r0 against CLT 14.2).

[fxcoudert]

That's the issue: the compiler should be used with the same SDK they were built against. We don't have a systematic way to support running against a different SDK than the build one.

[iains]

That's the issue: the compiler should be used with the same SDK they were built against. We don't have a systematic way to support running against a different SDK than the build one.

so should I do the noted back port or is it not needed (assuming the criterion above is applied)?

[fxcoudert]

Commit 1dc143181550573c9c902fb7a3b495e9b409d0b0 should be backported to all branches. But I don't think this bug is related.

[simonjwright]

Homebrew's GCC 14.2 (installed just now) manages to run just fine on a Sonoma system with CLT 16.0 and also with CLT 15.3.

It looks to me as though HB builds GCC using the FSF distribution + patches derived from Iain's distribution here, is that right? I just applied those patches of FSF GCC 14.2.0, built with CLT 15.3, but the result fails with CLT 16.0.

My interest is in supporting Ada users, and I need to minimise the difficulty they encounter in the macOS environment. Since raising this issue, I've of course advised people not to allow their machines to update to SDK 16 -- which isn't ideal.

[iains]

The fact is that SDKs from different Xcode releases can contain incompatible changes.

Since the process of building GCC (unfortunately) captures some information from the SDK into the build - we cannot (as things stand) guarantee that a build made with Xcode N will work with Xcode N-1 or N+1 (it might, but that's a matter of luck).

To make things as simple as possible for your end-users, I guess you could provide builds against Xcode 15 and 16, then they have to update their GCC if they update their Xcode?

Everything is fixable - if we had a team of folks working on GCC for macOS for their $dayjob, we could surely do better - - but that's a long way from reality at the moment ;)

[simonjwright]

To make things as simple as possible for your end-users, I guess you could provide builds against Xcode 15 and 16

Not sure how to do that. Trying to use a GCC built against CLT 15 to build a compiler against CLT 16 fails very early on in configure .. unable to build a C executable.

[iains]

To make things as simple as possible for your end-users, I guess you could provide builds against Xcode 15 and 16

Not sure how to do that. Trying to use a GCC built against CLT 15 to build a compiler against CLT 16 fails very early on in configure .. unable to build a C executable.

Do you have both the CLT installations available?
What is the --with-sysroot= configuration from the XC15 version?

[iains]

IFF you have Xcode CLT 16 installed, but you built the XC 15 one using --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOS.sdk ... then the following might work ...

on your configure line
CC="x86_64-apple-darwinNN-gcc --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOS14.sdk" CXX="x86_64-apple-darwinNN-g++ --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOS14.sdk"

Where NN is the relevant Darwin version for the bootstrap toolchain (i.e. the one built against the MacOS14 SDK in Xcode 15).

[iains]

The goal here is for the bootstrap toolchain to use the SDK it was built against - not to try and use the updated one installed in /Library/Developer/CommandLineTools/SDKs/MacOS.sdk

Under these circumstances it can be better to be specific about the SDK you are using.

Yes, we want it to be easier - which is why there's been an effort to reduce the dependency on fixincludes - but we cannot, unfortunately, get that dependency to 0 as things stand.

[simonjwright]

In /Library/Developer I have CommandLineTools-15.3 and CommandLineTools-16.0, and a symlink CommandLineTools selecting which one I want to use. It would be great if xcode-select --switch worked, but it doesn’t (it looks OK, but fails when you try to use it).

I’ll try your suggestion tomorrow when rested!

I suspect I may have to build an intermediate compiler that doesn’t have my adventurous specs fix, which I still haven’t posted a BZ about, sorry:

    --with-build-sysroot=$SDKROOT                                        \
    --with-sysroot=                                                      \
    --with-specs="%{!sysroot=*:--sysroot=%:if-exists-else($XCODE $CLT)}" \

[iains]

--with-build-sysroot=$SDKROOT   

At the risk of being repetitive - this is known to be broken - in particular, w.r.t. fixincludes https://gcc.gnu.org/bugzilla/show_bug.cgi?id=7988

[saschasc]

That's the issue: the compiler should be used with the same SDK they were built against. We don't have a systematic way to support running against a different SDK than the build one.

Is this documented somewhere when using the precompiled gcc versions from gcc? Currently, I am also struggling with compiling gRPC. I don't want and will hijack this thread ;-). But this detail would interest me if this is documented somewhere.

[simonjwright]

IFF you have Xcode CLT 16 installed, but you built the XC 15 one using --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOS.sdk ... then the following might work ...

on your configure line CC="x86_64-apple-darwinNN-gcc --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOS14.sdk" CXX="x86_64-apple-darwinNN-g++ --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOS14.sdk"

Where NN is the relevant Darwin version for the bootstrap toolchain (i.e. the one built against the MacOS14 SDK in Xcode 15).

This worked just fine.

• I used the gcc-14-2-darwin branch at commit 848f27a (actually, that was after a pull --rebase: the commit date was Tue Sep 24 16:11:14 2024 -0400),
• I had to build an intermediate compiler against CLT 15.3 according to your first suggestion, using--with-sysroot - my normal build, using --with-specs (as above) means that any --with-sysroot on the command line is overridden by a later one derived from the --with-specs option.
• I didn't need to use CC="aarch64-apple-darwinNN-gcc; I just used a full path, makes for a long command line but did the job.

You won't be surprised to learn that this compiler fails against CLT 15.3!

gcc hello.c
hello.c: In function ‘main’:
hello.c:6:3: error: implicit declaration of function ‘printf’ [-Wimplicit-function-declaration]
    6 |   printf("hi!\n");
      |   ^~~~~~
hello.c:3:1: note: include ‘<stdio.h>’ or provide a declaration of ‘printf’
    2 | #include <string.h>
  +++ |+#include <stdio.h>
    3 | 
hello.c:6:3: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch]
    6 |   printf("hi!\n");
      |   ^~~~~~
hello.c:6:3: note: include ‘<stdio.h>’ or provide a declaration of ‘printf’

[iains]

You won't be surprised to learn that this compiler fails against CLT 15.3!

gcc hello.c
hello.c: In function ‘main’:
hello.c:6:3: error: implicit declaration of function ‘printf’ [-Wimplicit-function-declaration]
    6 |   printf("hi!\n");
      |   ^~~~~~
hello.c:3:1: note: include ‘<stdio.h>’ or provide a declaration of ‘printf’
    2 | #include <string.h>
  +++ |+#include <stdio.h>
    3 | 
hello.c:6:3: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch]
    6 |   printf("hi!\n");
      |   ^~~~~~
hello.c:6:3: note: include ‘<stdio.h>’ or provide a declaration of ‘printf’

actually, that might be the one fixed by the pending backport - don't know if I'll get to it today - still doing the weekly rebases.

[iains]

I've now pushed that back port too - tested on macOS 12 only so far - so not making a release just yet - but if you want to test if the latest allows you some more flexibility ...

[iains]

where are we with this issue?

AFAIK I have now applied the relevant back ports - and just pushed a fix for another GCC/clang incompatibility.

[iains]

fixed in gcc-14.2-darwin-r2

[ilg-ul]

gcc-14.2-darwin-r2

Hi Iain, @iains

I did a new xPack GCC build with r2. The x64 binaries are fine (built on 10.14.6 with CLT 10.3 and MACOSX_DEPLOYMENT_TARGET=10.13, tested on macOS 12.7.6 with CLT 14.2), but unfortunately the arm64 binaries still fail when running in a newer environment (my build machine is macOS 11.7 with CLT 13.2, and the GitHub test machine is a 14.7.1 with Xcode 15.4).

The error is the well known stdio.h mismatch:

In file included from simple-hello-printf.c:10:
/Users/runner/work/gcc-xpack/gcc-xpack/build-assets/build/darwin-arm64/aarch64-apple-darwin23.6.0/tests/xpack-gcc-14.2.0-2/lib/gcc/aarch64-apple-darwin20.6.0/14.2.0/include-fixed/stdio.h:80:14: error: expected ';' before 'extern'
   80 | __BEGIN_DECLS
      |              ^
      |              ;
   81 | extern FILE *__stdinp;
      | ~~~~~~        
/Users/runner/work/gcc-xpack/gcc-xpack/build-assets/build/darwin-arm64/aarch64-apple-darwin23.6.0/tests/xpack-gcc-14.2.0-2/lib/gcc/aarch64-apple-darwin20.6.0/14.2.0/include-fixed/stdio.h:81:8: error: unknown type name 'FILE'
   81 | extern FILE *__stdinp;
      |        ^~~~
/Users/runner/work/gcc-xpack/gcc-xpack/build-assets/build/darwin-arm64/aarch64-apple-darwin23.6.0/tests/xpack-gcc-14.2.0-2/lib/gcc/aarch64-apple-darwin20.6.0/14.2.0/include-fixed/stdio.h:79:1: note: 'FILE' is defined in header '<stdio.h>'; this is probably fixable by adding '#include <stdio.h>'
   78 | #include <_stdio.h>
  +++ |+#include <stdio.h>
   79 | 
...

I read your comments about the lack of resources for fixing this. Unfortunately, apart from testing, there is not much I can do to help you.

If your final resolution is that there is nothing to do to fix the arm64 GCC such that builds performed on 11.7 (+CLT 13.2) to run on later environments, I'm afraid all I can do is to document it as unusable and move forward. :-(

[cjones051073]

gcc-14.2-darwin-r2

Hi Iain, @iains

I did a new xPack GCC build with r2. The x64 binaries are fine (built on 10.14.6 with CLT 10.3 and MACOSX_DEPLOYMENT_TARGET=10.13, tested on macOS 12.7.6 with CLT 14.2), but unfortunately the arm64 binaries still fail when running in a newer environment (my build machine is macOS 11.7 with CLT 13.2, and the GitHub test machine is a 14.7.1 with Xcode 15.4).

The error is the well known stdio.h mismatch:

In file included from simple-hello-printf.c:10:
/Users/runner/work/gcc-xpack/gcc-xpack/build-assets/build/darwin-arm64/aarch64-apple-darwin23.6.0/tests/xpack-gcc-14.2.0-2/lib/gcc/aarch64-apple-darwin20.6.0/14.2.0/include-fixed/stdio.h:80:14: error: expected ';' before 'extern'
   80 | __BEGIN_DECLS
      |              ^
      |              ;
   81 | extern FILE *__stdinp;
      | ~~~~~~        
/Users/runner/work/gcc-xpack/gcc-xpack/build-assets/build/darwin-arm64/aarch64-apple-darwin23.6.0/tests/xpack-gcc-14.2.0-2/lib/gcc/aarch64-apple-darwin20.6.0/14.2.0/include-fixed/stdio.h:81:8: error: unknown type name 'FILE'
   81 | extern FILE *__stdinp;
      |        ^~~~
/Users/runner/work/gcc-xpack/gcc-xpack/build-assets/build/darwin-arm64/aarch64-apple-darwin23.6.0/tests/xpack-gcc-14.2.0-2/lib/gcc/aarch64-apple-darwin20.6.0/14.2.0/include-fixed/stdio.h:79:1: note: 'FILE' is defined in header '<stdio.h>'; this is probably fixable by adding '#include <stdio.h>'
   78 | #include <_stdio.h>
  +++ |+#include <stdio.h>
   79 | 
...

I read your comments about the lack of resources for fixing this. Unfortunately, apart from testing, there is not much I can do to help you.

If your final resolution is that there is nothing to do to fix the arm64 GCC such that builds performed on 11.7 (+CLT 13.2) to run on later environments, I'm afraid all I can do is to document it as unusable and move forward. :-(

Can you not update your build machine to a newer OS ? macOS11 is a bit of an odd version to stick with, particularly on an ARM machine - the later OS versions improve arm support quite a bit...

[ilg-ul]

Can you not update your build machine to a newer OS ?

Unfortunately not :-(

xPack GCC is one of the 20+ projects that I build, and all binaries are based on older OSes (Ubuntu 18 for Linux, macOS 10.13 for x64 and macOS 11.7 for arm64). I have only one M1 Mac Mini, and it'll stay on 11.7 for the foreseeable future.

It is a pity that only macOS GCC has this problem, all other projects built on older machines run just fine on recent versions.

[ilg-ul]

My previous statement that the x64 GCC is functional is not true.

I paid one month of GitHub Team to get access to all GitHub runners, and so I was able to run tests on more versions.

It looks like there is no difference between x64 and arm64, both fail on macOS 14 and 15.

To conclude, with gcc-14.2-darwin-r2:

• the x64 binaries built on macOS 10.14.6 with CLT 10.3 and MACOSX_DEPLOYMENT_TARGET=10.13
  • pass on macOS 10, 11, 12, 13
  • fail on
    • macOS 14.7.1, Xcode 15.4, Build version 15F31d, com.apple.pkg.CLTools_Executables version: 16.1.0.0.1.1729049160, clang-1500.3.9.4
    • macOS 15.0.1, Xcode 16.0, Build version 16A242d, com.apple.pkg.CLTools_Executables version: 16.0.0.0.1.1724870825, clang-1600.0.26.3
• the arm64 binaries built on macOS 11.7.1 with CLT 13.2
  • pass on macOS 11, 13 (previous builds of r1 passed on Travis macOS 12.6, I guess 12 is ok with r2 too)
  • fail on
    • macOS 14.7.1, Xcode 15.4, Build version 15F31d, com.apple.pkg.CLTools_Executables version: 16.1.0.0.1.1729049160, clang-1500.3.9.4
    • macOS 15.0.1, Xcode 16.0, Build version 16A242d, com.apple.pkg.CLTools_Executables version: 16.0.0.0.1.1724870825, clang-1600.0.26.3

The Xcode version was obtained with xcodebuild -version and the CLT version with pkgutil --pkg-info=com.apple.pkg.CLTools_Executables.

I'm a bit confused why on macOS 14.7.1 CLT is 16.1 and Xcode is 15.4.

The two runs are:

• https://github.com/xpack-dev-tools/gcc-xpack/actions/runs/11836182813
• https://github.com/xpack-dev-tools/gcc-xpack/actions/runs/11842511558

I'm running out of ideas of what else I can do to help. :-(

@iains, any conclusion to this? Any chance to find a solution, or building GCC on an older version of macOS is a lost struggle?

[iains]

I regularly build GCC [x86_64 and arm64] on macOS 11, 12, 13 and 14 (with the relevant Xcode CLT for each)

As we have noted - there is no easy solution for the fix includes incompatibilities between CLT versions (sometimes). However a compiler version for (e.g.) macOS11 should still be able to build binaries that run on macOS 12..14 .. after all you do not expect your existing apps to stop working when you update the OS, do you?

That is, I would expect you can build perfectly functional compilers on all the versions where an earlier version is capable of building binaries that will run on a later one - and a later one is able to use -mmacosx-version-min= as needed.

What is not currently and without significant effort supported is a compiler built with one SDK using a different SDK.

Again, I would expect you can build working compilers for each platform version - maybe that's a pain - but after all you cannot run Xcode for macOS N on some arbitrary earlier macOS .. there are some constraints even there.

limited hardware means I don't have a setup for macOS15 yet..

... because of WG21 commitments I will not be able to poke at this for the next couple of weeks, I'd be very happy to make this my $dayjob and then we could try these extra improvements .. but until then ..

[ilg-ul]

However a compiler version for (e.g.) macOS11 should still be able to build binaries that run on macOS 12..14 .. after all you do not expect your existing apps to stop working when you update the OS, do you?

Well, I have such a program built on macOS 11, which runs perfectly fine on macOS 11, 12 & 13. After I upgraded to 14 & 15, it failed.

This program is called... gcc :-(

That is, I would expect you can build perfectly functional compilers on all the versions where an earlier version is capable of building binaries that will run on a later one - and a later one is able to use -mmacosx-version-min= as needed.

My build machine is 11.7 and I use -mmacosx-version-min=11.0. So far all other 20+ projects had no problems with that, and run just fain up to macOS 14 (probably macOS 15 too, I had no way to test them yet).

What is not currently and without significant effort supported is a compiler built with one SDK using a different SDK.

Yes, this seems to be a big challenge for GCC. :-(

I will not be able to poke at this for the next couple of weeks

For now I'll update the documentation to mention that the binaries work up to macOS 13, and switch all my test to clang.

If, before the next major GCC release, you'll find a solution to make a GCC built on macOS 11 run properly on macOS 14 & 15, I'll gladly adopt it, otherwise I have no choice but discontinue the GCC binaries for macOS. :-(

[iains]

What is not currently and without significant effort supported is a compiler built with one SDK using a different SDK.

Yes, this seems to be a big challenge for GCC. :-(

That's not the full situation - IFF the SDKs did not contain headers that needed correction, we would not have this issue. Since we cannot change the SDKs .. we have no choice but to work around issues.

Anyway, you are free to do whatever you wish - the compiler will work within its constraints.

[ilg-ul]

Since we cannot change the SDKs .. we have no choice but to work around issues.

I know it is different, but didn't clang manage to work around these issues?

[iains]

Since we cannot change the SDKs .. we have no choice but to work around issues.

I know it is different, but didn't clang manage to work around these issues?

It does not have to work around some of the issues - because the extensions it supports (either accidentally or intentionally) like (for ex) using C++ features in C code, are obviously compatible with the SDK headers.

However - the following still fires on clang.
Why is this a problem? Well the decision to make several of the stdio functions 'deprecated` is unilateral to Apple - these functions are (a) not deprecated by any extant C/C++ standard nor by Posix (b) they are very widely used in Open Source code.

cat t.c:

#include <stdio.h>

int main ()
{
  char buf[128];
  sprintf (buf, "Hello, world\n");
  puts (buf);
  return 0;
}

FAOD: I am not disputing that the use of these functions can lead to code vulnerabilities - however, the deprecation should be opt-in - not forced on the end-user. When code is built with WError, this becomes a build fail (sure, there are work-arounds - but that involves changing every project that uses the compiler ... )

So we patch the header to make the deprecation opt-in.

$ /XC/16.1b2/CommandLineTools/usr/bin/clang -isysroot /XC/16.1b2/CommandLineTools/SDKs/MacOSX15.sdk -xc++ t.c
t.c:7:3: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations]
    7 |   sprintf (buf, "Hello, world\n");
      |   ^
/XC/16.1b2/CommandLineTools/SDKs/MacOSX15.sdk/usr/include/_stdio.h:274:1: note: 'sprintf' has been explicitly marked deprecated here
  274 | __deprecated_msg("This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
      | ^

[ilg-ul]

It does not have to work around some of the issues - because the extensions it supports (either accidentally or intentionally) like (for ex) using C++ features in C code, are obviously compatible with the SDK headers.

Ah, so the extra patches for GCC are needed only to avoid the clang extensions?
Does this mean that the only safe solution is to add the same extensions to GCC too?

However - the following still fires on clang. ... the stdio functions 'deprecated`

Good to know.

I am probably lucky, in the xPack project I compiled from sources about 70 open source projects and I don't remember encountering this issue.