Add universal prebuilt Mac package (x86_64 + arm64)

[kinke]

No description provided.

[kinke]

This seems like less trouble than anticipated; according to CI, it's fine on x86_64 hosts. Some quick smoke tests on actual Apple Silicon (cross-compiling & -linking to all 4 targets) would be appreciated; if you have time, maybe also checking the arm64 package and that it can target iOS too. Pinging @p0nce, @dnadlinger, @schveiguy.

CI artifact(s) downloadable from https://dev.azure.com/ldc-developers/ldc/_build/results?buildId=3437&view=artifacts&pathAsName=false&type=publishedArtifacts.

[dnadlinger]

Seems to work fine for building a Phobos Hello World on 12.2.1/arm64, for both arm64 and x86_64 targets (with the 1.29.0-git-788b8ef binaries from your comment).

[p0nce]

All tests on macOS Monterey 12.0.1, Xcode 13.0, M1
what I haven't tested = the Universal package on x86_64 host.

I'm testing intel-intrinsics in unitest-release mode, to avoid the -g bug in Monterey.

(Reference run with LDC 1.28 x86_64
dub test -a x86_64-apple-macos -b unittest-release => OK
dub test -a arm64-apple-macos -b unittest-release => OK)

LDC is Universal Binary

dub test -a x86_64-apple-macos -b unittest-release --compiler /Users/guill/Desktop/compilers/test/ldc2-788b8ef0-osx-universal/bin/ldc2 => OK

dub test -a arm64-apple-macos -b unittest-release --compiler /Users/guill/Desktop/compilers/test/ldc2-788b8ef0-osx-universal/bin/ldc2 => OK

dub test -a x86_64-apple-ios -b release --compiler /Users/guill/Desktop/compilers/test/ldc2-788b8ef0-osx-universal/bin/ldc2 => OK, successfully link and runs, crash the with dyld[1918]: DYLD_ROOT_PATH not set for simulator program which sounds alright.

dub test -a arm64-apple-ios -b release --compiler /Users/guill/Desktop/compilers/test/ldc2-788b8ef0-osx-universal/bin/ldc2 => OK, successfully link and crash the unittests with error code 9, probably my fault

So yeah I guess it can become the one and only LDC on macOS :)
Also tests the now obsolete packages:

LDC arm64 package

dub test -a x86_64-apple-macos -b unittest-release --compiler /Users/guill/Desktop/compilers/test/ldc2-788b8ef0-osx-arm64/bin/ldc2 => OK, link failure as expected, as arm64 LDC doesn't have x86_64 target.

dub test -a arm64-apple-macos -b unittest-release --compiler /Users/guill/Desktop/compilers/test/ldc2-788b8ef0-osx-arm64/bin/ldc2
=> link failure, not sure why, not a concern if universal supersedes that

ld: warning: ignoring file /Users/guill/Desktop/compilers/test/ldc2-788b8ef0-osx-arm64/lib/libphobos2-ldc.a, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
ld: warning: ignoring file /Users/guill/Desktop/compilers/test/ldc2-788b8ef0-osx-arm64/lib/libdruntime-ldc.a, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
ld: warning: ignoring file .dub/build/intel-intrinsics-test-library-unittest-release-posix.osx.darwin-aarch64.arm_hardfloat-ldc_v1.29.0-git-788b8ef-6D0F504A78D2DED12452D86E33078DC9/intel-intrinsics-test-library.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
Undefined symbols for architecture x86_64:
  "_main", referenced from:
     implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error: /usr/bin/cc failed with status: 1

LDC is x86_64

dub test -a x86_64-apple-macos -b unittest-release --compiler /Users/guill/Desktop/compilers/test/ldc2-788b8ef0-osx-x86_84/bin/ldc2 => OK

dub test -a arm64-apple-macos -b unittest-release --compiler /Users/guill/Desktop/compilers/test/ldc2-788b8ef0-osx-x86_84/bin/ldc2 => OK, link failure, but expected since x86_64 package can't target arm64 anymore.

[schveiguy]

Well, this isn't working for me on OSX 12.3.1. Specifically, the -g flag doesn't work still (get linker errors).

Note also, with a previous LDC, if I set the environment variable: MACOSX_DEPLOYMENT_TARGET=11 it worked. In this version, that workaround doesn't work. So I would not be able to use this binary.

The specific error is:

steves@MacBook-Pro testd % ~/Downloads/osx-universal/ldc2-788b8ef0-osx-universal/bin/ldc2 -g testbadlink.d
ld: warning: pointer not aligned at address 0x100008032 (anon + 50 from testbadlink.o)
ld: unaligned pointer(s) for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error: /usr/bin/cc failed with status: 1

See #3864 for more details.

[kinke]

Thx for testing. - This doesn't affect/fix #3864 at all; if there are changes compared to before, that's either because of LLVM 14 or some further Xcode changes or whatever.

[schveiguy]

@jacob-carlborg any insights here? You were the one I think who figured out the workaround last time.

[kinke]

Off-topic, but there is another change for this CI artifact here compared to the latest releases - the macOS image for building LDC, druntime and Phobos was bumped from 10.15 to 11.x, and so the Xcode SDK version with it.

MACOSX_DEPLOYMENT_TARGET during the build was raised from 10.9 to 10.12 for the x86_64 package; still at 11.0 for the arm64 package.

One workaround is adding -preserve-dwarf-line-section=false to the cmdline / ldc2.conf file, implying no file/line infos in druntime exception backtraces. A debugger would presumably still offer those infos.

[p0nce]

10.12 as target is more than OK, because once notarized a lot of software doesn't launch on 10.11 anyway except if you use a special flag.

[jacob-carlborg]

any insights here? You were the one I think who figured out the workaround last time.

Unfortunately no. I think the best way forward, assuming the -preserve-dwarf-line-section=false flag works, is to read the debug info from the object files instead of the executable, that's what the debugger is doing. Debug info can also be compiled to its own .dSYM file for distribution. The best would be if all three methods were supported by druntime. @Geod24 is working on this, IIRC.

[kinke]

@dnadlinger: Can you confirm Guillaume's trouble with the arm64 package? The cc invocation apparently defaulting to x86_64, not arm64 as expected. ldc2.conf hasn't changed for the macOS-arm64 target (edit: still no explicit -target arm64-apple-macos), so I'd have expected according bug reports for the existing arm64 package if linking wouldn't work.

Please also test the arm64-apple-ios12.0 target with the arm64 package. And also the default targets for both universal and arm64 packages, without any -mtriple (should obviously default to macOS-arm64).

[jacob-carlborg]

I've tried all combinations:

Host: x86_64 and arm64
Targets: x86_64-ios, x86_64-macos, arm64-macos, arm64-ios and default target
Packages: universal, x86_64 and arm64

Everything that is expected to link does link.
All binaries that are expected to run do run. I did not bother trying to run the iOS binaries in simulator or on device.

I could not reproduce Guillaume's issue with the arm64 package.

[jacob-carlborg]

Is the intention that the universal package will replace the other two?

[p0nce]

I didn't used the dub from the archive, so maybe related. Anyway, I don't think it's important given the new package and its reach.

[schveiguy]

Oh yeah! I have used in the past an X86_64 dub while specifying the ldc2 compiler, and it doesn't work properly. And thinking about it, I can't imagine why that is true, unless rosetta does something weird when executing the linker from an x86 application?

[kinke]

Excellent Jacob, thx a lot!

Is the intention that the universal package will replace the other two?

Nope. The 2 base packages are needed for merging anyway, so keeping on uploading them to GitHub is no hassle (excluding them would be more work).

[dnadlinger]

@kinke:

Can you confirm Guillaume's trouble with the arm64 package?

Sorry, I'm a bit out of the loop here – what exactly is the issue? It appears to do what it should on a standard macOS 12.2 arm64 install (MacBook Pro M1 Max, Xcode via the App Store).

~ ❱ ~/Downloads/ldc2-788b8ef0-osx-universal/bin/ldc2 hello.d
~ ❱ file hello
hello: Mach-O 64-bit executable arm64
~ ❱ ./hello
Hello world!
~ ❱ rm hello
~ ❱ ~/Downloads/ldc2-788b8ef0-osx-arm64/bin/ldc2 hello.d
~ ❱ file hello
hello: Mach-O 64-bit executable arm64
~ ❱ ./hello
Hello world!

[schveiguy]

what exactly is the issue?

Almost certainly, the issue was with using an x86_64 dub executable. For what reason, I'm not sure, but I had the exact same experience a few months ago. Switching to the ldc2-provided dub package works (maybe @p0nce can confirm it works for him?)

[p0nce]

If it work for everybody but for me, it is certainly DUB. I'm not home and I tested already so... there is probably no issue.

[dnadlinger]

macOS will choose to run the x86_64 version of a child process binary if the (any?) parent is x86_64, so that makes some sense. Should we explicitly be passing the architecture on the linker command line to avoid relying on the default target, if that is what is going on?

[schveiguy]

macOS will choose to run the x86_64 version of a child process binary if the (any?) parent is x86_64

Oooh! that explains it. Dub was running ldc2 for linking, and when I copied the exact command, it worked (because now, I'm running it from the shell). I don't think dub expects to pass arch to the linker.

Is there a doc explaining this somewhere?

[kinke]

macOS will choose to run the x86_64 version of a child process binary if the (any?) parent is x86_64, so that makes some sense.

Thx for the explanation.

Should we explicitly be passing the architecture on the linker command line to avoid relying on the default target, if that is what is going on?

I don't think so - the universal package specifies the clang triple for all 4 targets, just the arm64 package doesn't for its default target (edit: just like the x86_64 package). This 'problem' with an external x86 dub should also apply to universal LDC itself then, running the x86 ldc2, wasting time for emulation, affecting the default LDC target without explicit dub arch etc. etc., and should thus be avoided anyway.