Migrate xcframework packaging from xcodebuild to SwiftPM

[brianmichel]

Description

Binary targets are a vital part of the SwiftPM ecosystem and currently any discussions about extending the format or expanding usage of this format (or xcframeworks) in general are limited due to the packaging tool being included as part of xcodebuild. Interestingly, computing the checksum for a binary target is included as part of the swift package suite of tools, but not the creation of said package.

I believe migrating this functionality from xcodebuild to SwiftPM would be advantageous as it could open up discussions to make SwiftPM even more useful on non-Darwin platforms that want to use Swift.

Expected behavior

I'd like to write something like the following:

swift package create-xcframework -framework <path> [-framework <path>...] -output <path>
or
swift package -create-xcframework -library <path> [-headers <path>] [-library <path> [-headers <path>]...] -output <path>

Which would create the framework similarly as it does when running the command from xcodebuild

Actual behavior

This command doesn't exist today and you're forced to run xcodebuild -create-xcframework on macOS.

Steps to reproduce

1. Launch a Linux or Windows installation that has Swift installed
2. Attempt to run either xcodebuild -create-xcframework ... or swift package create-xcframework ... and see that it doesn't work

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.9.0

Swift & OS version (output of swift --version && uname -a)

swift-driver version: 1.87.1 Apple Swift version 5.9 (swiftlang-5.9.0.128.108 clang-1500.0.40.1)
Target: arm64-apple-macosx14.0
Darwin zerocool.localdomain 23.0.0 Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:43 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T6000 arm64

[brianmichel]

I'd love to help with this, but I imagine the bulk of the work is on the xcodebuild side which isn't open source.

[brianmichel]

The more I've thought about this, I realize that we probably can't move this function out of xcodebuild due to being able to package objective-c based frameworks and libraries.

Maybe the larger question should be how to provide a better cross platform format that SPM would be happy with for these binary artifacts?

Additionally, does an xcframework allow for the inclusion of other executable resources like apps that be loaded as part of the framework itself, or not really?

[MaxDesiatov]

Additionally, does an xcframework allow for the inclusion of other executable resources like apps that be loaded as part of the framework itself, or not really?

Can you elaborate on what "loaded" means here? How would you expect executable binaries to be loaded? At run time or build time?

[neonichu]

If cross platform support is the main concern here, we already have the artifact bundle format for non-Apple platforms. I believe @MaxDesiatov was also working on adding support for that to library targets (today it is only available for executables).

[brianmichel]

Oh! Is there documentation for this format? Cross platform artifact sharing is the main problem I’m trying to mitigate without needing to use something like nuget which is available across all platforms we’re targeting.

…

On Mon, Oct 30, 2023 at 11:45 AM Boris Bügling ***@***.***> wrote: If cross platform support is the main concern here, we already have the artifact bundle format for non-Apple platforms. I believe @MaxDesiatov <https://github.com/MaxDesiatov> was also working on adding support for that to library targets (today it is only available for executables). — Reply to this email directly, view it on GitHub <#7035 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAPTL2QZVCSS5YGBBGA2MLYB7DQNAVCNFSM6AAAAAA6OTSSCCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBVGUYDANBSGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[MaxDesiatov]

Yes, the format is specified in SE-0305. Per the proposal it only supports executables. SE-0387 reuses this format for Swift SDKs.

I have a PoC for libraries in #6967, but that will require an implementation update, and an addition of a checking mechanism for libc/libc++ flavors and versions. And when all of that is ready in a PoC implementation, we'll have to pitch it on the forums before it goes anywhere.

[brianmichel]

Ah cool, so this is, unfortunately, not imminent, but seems like it's actively being worked on. I'm just trying to figure out how much time to invest in a different solution vs doing what I can to help this along. Any thoughts @MaxDesiatov ?

[brianmichel]

Ah cool, so this is, unfortunately, not imminent, but seems like it's actively being worked on. I'm just trying to figure out how much time to invest in a different solution vs doing what I can to help this along. Any thoughts @MaxDesiatov ?

Also let me know however I can help, I'm (selfishly) very interested in this!

[MaxDesiatov]

The main roadblock with this work is that it has to be a solution that works for all platforms that Swift supports, but only Darwin platforms support ABI stability, and only Darwin platforms can have deployment targets cleanly specified.

Just looking at Linux, we have multiple edge cases to consider:

1. What happens if you build a library artifact with a current Glibc version, but try to link on a platform that has older Glibc?
2. What happens if you build with Glibc, but link on a platform that uses Musl?
3. Repeat 1 and 2 for libstdc++ and libc++ combinations.
4. Repeat 1-3 for an arbitrary dynamically-linked systemLibrary that can be declared in Package.swift.
5. How can we detect a version of a system dependency in general, which we can't control and which is controlled by a system package manager?
6. Answer corresponding questions for the Windows target.

When we have clearly specified answers to these questions and mechanisms to implement them (help with that is appreciated), then PoC work can proceed.

[brianmichel]

Wow, lots of edge cases! Apologies if any of the below is uninformed and there are already really good reasons to not do things that way, I'm just exploring this space myself :)

For questions, 1 through 3, would we consider implementing something like a requires/depends field similar to what .deb files are typically created with? This would allow the author to specify the needed libraries/runtimes to successful use the package.

For number 4 I'm not quite sure what the what this scenario is so I don't have any ideas sadly!

For number 5, would we not want to ask the user what package management system they use, and shell out to that to find things like installed version, system paths, etc? Apologies if this is a bit naïve, but I figured I'd ask to better orient myself around the goals y'all have with SPM.

[kelteseth]

@MaxDesiatov We need this feature for the Godot swift bindings for Windows (and Linux). Building them takes about 13 minutes on my 16 core machine.

MSVC binaries are generally compatible within the Visual Studio 2015 and later versions:

1. Binary compatibility: Visual Studio 2015 (v140) and later versions (2017/v141, 2019/v142, 2022/v143) are binary-compatible. This means libraries built with one version can be used with applications built by another, without recompilation.

2. Linking rule: The linker should only work with inputs built by a toolset that is the same version or earlier than itself. For example, you can link VS 2015 and VS 2022 binaries, but you must use the VS 2022 linker or later.

https://learn.microsoft.com/en-us/cpp/porting/binary-compat-2015-2017?view=msvc-170

[xtyxtyx]

I tried to migrate #6967 to the current main branch and created an example project that works across macOS, Linux, and Windows. You can find the code and build instructions here for those interested.

In my case, I need to use Skia in a Swift project. Building Skia requires specialized tools such as depot_tools, gn, ninja, and various Python scripts, and the build process often more than an hour, which both makes it impractical to build Skia with SPM. Therefore, the ability to use prebuilt Skia binaries is crucial for my project.

Additionally, since most of Skia’s dependencies are statically linked, perhaps the mechanism for ensuring the compatibility of dynamic libraries isn’t as critical in such cases.