Fix building on OSX (script, installation)

[karwa]

We currently cannot build Foundation on OSX using the build script. Running the swift build script on OSX with --Foundation will not succeed. There is an Xcode project, but to use it we already need the swift toolchain installed, and it generates a Framework where we might prefer a dylib.

As part of this change, the name of the Mac build of the open-source Foundation module has had to change - it obviously can't be 'Foundation', but it can't be 'SwiftFoundation' either as our dylib would conflict with the native SDK overlay libswiftFoundation.dylib (only differing by case). With that in mind, I've tentatively named it 'PortableFoundation'.

Also, we've had to change the name of the environment variable the swift build script uses to tell us about the swift build location; currently it uses SDKROOT, but xcrun uses that variable and the result is we can't find the actual SDK location on OSX. I've renamed it SWIFT_SDKROOT, so we'd need a corresponding change to swift's build-script-impl

[parkera]

I'm not really a fan of the name PortableFoundation but I'm not immediately sure of what else it should be called.

Really, this build product is something that is the opposite of PortableFoundation - it's a version of corelibs-foundation that is built temporarily for OS X.

[karwa]

yeah I'm not sure what to name it.

It is actually portable; in cases where you don't need Objective-C it gives you a modern Foundation API even if Apple never shipped a new version for your OS. Let's imagine I'm on an older OSX and I want to install some command-line utility that uses newer Foundation APIs; I might edit the package and switch it to use PortableFoundation. Maybe I'm writing such a tool, and I want to make sure I'm writing against the full, modern API (e.g the new NSCalendar API) instead of being limited by @available constraints. In that case, you might decide to write for PortableFoundation even though you're on OSX. System Foundation requires system version checks, deployment targets and so on, while PortableFoundation requires none of those things. I would even argue in favour of installing it by default in Swift toolchains and to recommend everybody doing cross-platform work to target that instead (or you may find it that, while it works for you, the #os(OSX) branch will suddenly be full of potentially-unavailable calls and may just fail to compile.

Not only that, but I couldn't actually find a clean way to tell the Swift compiler which minimum OSX version we support. The only way I could find is with the version at the end of the triple, and that has problems - by default, even on OSX 10.11.4, a plain swift compile will only target x86-apple-macosx10.9. So cross-platform package developers will need to explicitly set a target, and all swift code that uses the package must do the same. So we're requiring them to have an incredible amount of intricate knowledge about the history of OSX APIs, none of which shows up on their platforms and all of which could be avoided by using PortableFoundation instead of the system one.

Building this project for Macs is not just a curiosity which is helpful for development/debugging; in some cases this library may be superior to what you have from Apple. Sure, it's lacking Objective-C, but for lots of people that's just fine. It's a bit galling that everybody else has a fully modern, updatable Foundation library with no availability constraints, but Mac users have to use a library baked in to the OS whenever it happened to be made. I'm not just in favour of renaming SwiftFoundation; I'm in favour of shipping it by default with every copy of Swift, especially on OSX.

I'd support renaming Apple's non-portable Foundation to 'ObjCFoundation' or 'MacFoundation' or something, but that'll obviously never happen...

[parkera]

It is not our intent to ship this on Darwin. For one thing, if you try to use any other framework in the SDK with this version of Foundation you will probably crash, since we changed the layout of CFTypeRef.

[ianpartridge]

I support the intent of this PR - I would like to be able to build and run Swift applications using swift-corelibs-foundation on OS X. People with cross-platform codebases which depend on Foundation APIs will benefit from being able to test and run against both implementations. Thank you for working on this!

[karwa]

Then the 'proper' solution lies in shipping this and flagging those frameworks as incompatible, so that importing them in the same module produces a compiler warning.

As it stands, the Foundation in this repo is portable for all platforms except Darwin-based ones. Considering that this is from Apple, and the intent of the project, that's a bit of a facepalm IMHO.

Supposedly cross-platform code will be littered with availability checks which only make sense on Darwin. Not only that, but as a linux developer you won't know which APIs are available on which versions of OSX or not, so you're going to have to spend an inordinate amount of time researching every single Foundation API you use on developer.apple.com. That's not realistic.

[modocache]

One question: swift-corelibs-xctest builds swift-corelibs-foundation as part of its CI build, on both OS X and Linux. On OS X, we import SwiftFoundation. This system has been in place since swiftlang/swift#1556 (March 6). How will this pull request affect our test setup?

Also, not that anyone asked, but here's my two cents:

• swift-corelibs-xctest only supports Linux. I have no intent of shipping anything that works on OS X.
• However, corelibs-xctest does provide an .xcworkspace that is capable of building and running the tests on OS X, purely to benefit contributors who prefer to write patches on an OS X environment. I don't know what build products the .xcworkspace produces, I don't really care. :)
• To ensure that .xcworkspace continues to function as a nice tool for contributors, we test it on CI. Running swift/utils/build-script --xctest --test invokes xcodebuild SWIFT_EXEC=/path/to/the/build/swiftc. This has caught OS X build errors not only for our .xcworkspace on OS X, but also for Foundation.xcodeproj in the past.
• Having built the system I describe above, I naturally am in favor of doing the exact same thing for swift-corelibs-foundation. :)

Supposedly cross-platform code will be littered with availability checks which only make sense on Darwin.

This is exactly what the swift-corelibs-* projects are hoping to avoid. Their goal is to provide a 100% identical API to the Apple frameworks, and a functionally identical implementation.

[karwa]

I actually got around to this by accident while working on changes to support cross-compiling. I haven't gotten to xctest yet, but it should just be a name change.

This is exactly what the swift-corelibs-* projects are hoping to avoid. Their goal is to provide a 100% identical API to the Apple frameworks, and a functionally identical implementation.

That works fine if you only care about porting Darwin code (with checks already) to other platforms. Code written for those other platforms won't work and continue to work. I suspect that developers who aren't centered on OSX will take your attitude, and either shun the project entirely for not being cross-platform enough, or just ditch OSX compatibility. Very, very few people are going to decide that it's worth it to go through that fuss.

Imagine that - a rich community of people developing against Foundation, and hardly any of it working on OSX. That's the absurd situation I'm worried might develop.

[modocache]

I suspect that developers who aren't centered on OSX will take your attitude, and either shun the project entirely for not being cross-platform enough, or just ditch OSX compatibility.

I'd like to think that I've made technical decisions within the constraints of the project as presented when it was open-sourced--no "attitude" here! :) I think the issue is complex. If you feel strongly about it, please feel free to create a JIRA task or send a pull request to swift-corelibs-xctest -- I really only have marginal knowledge of swift-corelibs-foundation.

That works fine if you only care about porting Darwin code (with checks already) to other platforms.

I think the goal of the swift-corelibs-* projects is to eliminate the "checks" you mention. But I do agree that swift-corelibs-xctest really is designed to port Darwin code to other platforms. In fact, several members of the Apple core team have commented on the mailing lists that the XCTest API is not suited to Swift, and I agree. If Apple were to build a testing framework from scratch for Swift, how likely is it that they would use reflection to discover test cases at runtime? How likely would they choose to tie the concept of a test suite to the concept of a "bundle"? I think not very likely.

Still, swift-corelibs-xctest has ported the Apple API faithfully to Linux. Not because it's the best cross-platform API, but because it's the API that exists on Darwin. The idea is that tests written on Darwin don't have to be modified or rewritten in order to run them to Linux.

I think there's room for people to build a better cross-platform testing framework, one that doesn't support tests already written in Apple XCTest. I consider swift-corelibs-xctest to be "a Darwin testing framework ported to Linux" more than I think of it as a "cross-platorm testing framework". I genuinely encourage anyone who wishes to "shun the project entirely for not being cross-platform enough" (and in a sense, I agree!!) to build a better testing framework, and to chime in on testing-related evolution proposals like swiftlang/swift-evolution#51.

[parkera]

Maybe instead of PortableFoundation we call it CoreLibsFoundation (not to be confused with CoreFoundation).

[phausler]

I am not certain this is the right move here. It has some consequences that might sign us all up for things that would tie our hands for fixing things in the future. It might be a good idea to break this up into more than one pull request so we can consider the full meaning individually

[karwa]

We'd still need to ship on Darwin - either bundled with Swift, or via some kind of package manager...

I didn't create the bug to be political about the direction of the project - I just happened to notice the availability issue, and it points to the underlying problem. It's pretty fundamental - code written on Linux using this library is not source compatible with whichever Foundation happens to ship on whichever version of OSX you happen to be compiling from. Maybe it is the other way around - if so, that's great for Linux; you can build and run stuff that used to be Mac/iOS-only.

For Mac users, life would be much less pleasant. If you're not on the latest OSX, your Foundation won't match this project, and some of the symbols used by packages you download might not exist. We have a mechanism to handle it (@available attributes), but linux developers won't have a Foundation module flagged with those attributes for OSX, so they won't even know who they're tripping up, it won't be enforced by their compiler, and having to research which calls were introduced when is an unconscionable amount of work. Linux developers would be quite right to forget about targeting Darwin platforms - after all, their users can upgrade their machines without any of these problems.

It happens all the time. NSNumberFormatterStyle.OrdinalStyle got added in OSX 10.11. Let's imagine .PigLatinStyle gets added in 10.12 (or whenever, in the future), and of course, it's in this library too. Now I write a package which uses .PigLatinStyle, and it runs on Linux because it's using this library; it also runs on OSX 10.12 (with the correct -target compile flag), because the symbol exists in the native Foundation, but it'll fail on 10.11. Those users can forget about using my awesome package.

If you only ever compile the project for OSX in Xcode, you won't notice it because it always compiles with the target x86_64-apple-macosx10.11, which basically formalises what I've said above about Mac users being "frozen in time". Originally I built the library like that too, manually specifying a target triple like Xcode does; but this sets a very high minimum deployment target and forces all importing code to do the same. You can't even use it in a REPL without manually entering a triple - by default, Swift works with a 10.9 deployment target and refused to load the module.

[parkera]

I think this idea needs some more formulation before we decide to take it as a patch. Let's open a ticket in JIRA (if this is still important) and continue the discussion there.