[SR-9384] Figure out why Foundation integration test for compiling on Linux with -static-stdlib fails

[gottesmm]

Previous ID	SR-9384
Radar	None
Original Reporter	@gottesmm
Type	Task

Additional Detail from JIRA

Votes	0
Component/s	Foundation, libdispatch
Labels	Task
Assignee	None
Priority	Medium

md5: a5cc649f55665ecc6571bab8d1d28caa

relates to:

• SR-10625 tests for -static-*

Issue Description:

I committed this test thinking that this was working. I am going to disable this and send it over to compnerd/spevans who were the ones messing with this I believe.

From the bot:

"/home/buildnode/jenkins/workspace/swift-PR-Linux@2/branch-master/buildbot_linux/none-swift_package_sandbox_linux-x86_64/usr/bin/swiftc" "-static-stdlib" "/home/buildnode/jenkins/workspace/swift-PR-Linux@2/branch-master/swift-integration-tests/test-static-stdlib/main.swift" "-o" "/home/buildnode/jenkins/workspace/swift-PR-Linux@2/branch-master/buildbot_linux/none-swift_package_sandbox_linux-x86_64/tests/test-static-stdlib/Output/test-static-stdlib.test.tmp/main"

And a selection of errors:

/usr/bin/ld.gold: error: /home/buildnode/jenkins/workspace/swift-PR-Linux@2/branch-master/buildbot_linux/none-swift_package_sandbox_linux-x86_64/usr/lib/swift_static/linux/libdispatch.a(apply.c.o): requires unsupported dynamic reloc 11; recompile with -fPIC

*01:36:40* /home/buildnode/jenkins/workspace/swift-PR-Linux@2/branch-master/buildbot_linux/none-swift_package_sandbox_linux-x86_64/usr/lib/swift_static/linux/libFoundation.a(Bundle.swift.o):Bundle.swift.o:function $s10Foundation6BundleC16bundleIdentifierSSSgvg: error: undefined reference to 'CFBundleGetIdentifier'

[gottesmm]

@spevans @compnerd thoughts?

[gottesmm]

Here is the build:

https://ci.swift.org/job/swift-PR-Linux/9316/

[gottesmm]

Disabled in thie PR: swiftlang/swift-integration-tests#45

[spevans]

When I was trying this out before I think I just used the -fPIC for libdispatch and it worked.

The 2nd error looks to be that CoreFoundation.a is not linked into libFoundation.a

[spevans]

I ran this test myself and there is also an error, undefined reference to strlcpy and getprogname. This is due to a lack of -lbsd.

We could try and fix up the auto link or alternatively make libdispatch use program_invocation_short_name and snprintf on Linux and remove the use of -lbsd completely.

[compnerd]

Cool, I think that fixing the PIC thing is easy. The linking ... well, that requires a bit more effort. The problem is that ELF doesn't have autolinking. I've added an extension to LLVM to add support for this to lld. We could hack this via the linker response file that we build.

[gottesmm]

I would love if swift on Linux used lld provided lld is ready (that is my only concern). It will enable us to use more advanced features and a higher likelihood of being able to push in features to the linker that we need on Linux without having to leave LLVM land.

[compnerd]

Created #434 for building libdispatch as PIC. Tweaking the CoreFoundation link is a second issue.

@gottesmm, I think that the biggest blocker to that is the fact that it requires updating the `update-checkout` and `build-script` monsters. I can tell you that it has been extensively tested and should be considered production ready for Linux (and resultantly, android) x86, x86_64, arm, aarch64. I am unsure if the Linux PPC, PPC64 targets have received as much testing. Windows x86, x86_64 should be considered production ready, the support for Windows arm be considered experimental, and Windows arch64 is missing.

@spevans, when you say that CoreFoundation is not linked into Foundation what do you mean? static libraries aren't really linked per se. Should we be including the target objects of CoreFoundation into Foundation?

[spevans]

Yes, before the CMake changes, libFoundation.a had the libCoreFoundation.a objects included:

$ nm swift-4.2-RELEASE-ubuntu18.04/usr/lib/swift_static/linux/libFoundation.a |grep ' CFHashBytes'
00000000000000f0 T CFHashBytes
                 U CFHashBytes
[...etc...]

$ nm swift-DEVELOPMENT-SNAPSHOT-2018-11-30-a-ubuntu18.04/usr/lib/swift_static/linux/libFoundation.a |grep ' CFHashBytes'
                 U CFHashBytes
[...etc...]                 

[gottesmm]

@compnerd I am fine in that case, but I think it is important that a discussion happens on swift-dev. Additionally, instead of conditionally cloning lld only on linux (like we are doing now with ICU), can we just always check it out.

It makes it so that I can clone on a Darwin host and then compile on a docker Linux much faster.

[spevans]

I tested the PIC change and added -lbsd into utils/gen-static-stdlib-link-args which generated the usr/lib/swift_static/linux/static-stdlib-args.lnk file and these two changes worked so I think merging the CoreFoundation objects into libFoundation.a is the last bit to do.

[spevans]

This looks to have been fixed by https://github.com/apple/swift-corelibs-foundation/pull/2850

I locally re-enabled the test disabled in swiftlang/swift-integration-tests#45 and it passed so I think this is now fixed.

[spevans]

PR to re-enable this test: swiftlang/swift-integration-tests#78

This should work now that -static-stdlib works on Linux