Update rules_spm to build fetched dependencies using Bazel instead of Swift Package Manager

[cgrindel]

Goals

• Fetch dependencies using Swift Package Manager (SPM)
• Generate Bazel build targets for the fetched dependencies
• Ensure that the fetched dependencies have typical Bazel dependency names (e.g. https://github.com/grpc/grpc-swift becomes com_github_grpc_grpc_swift)
• Provide a mechanism for external dependencies to have a custom Bazel dependency name.

Reasoning

• Address duplicate symbol errors when the client project and a client project dependency both load the same dependency
  • Depend on package but not its dependencies #148
  • Handle project that uses rules_spm and one of its dependencies also uses rules_spm #86
• Allow target platform compilation to work properly (e.g., build for iOS device on macOS)

Tasks

• Add support for pure Swift packages
• Add support for packages with C that do not have special settings (e.g. CSetting).
• Derive source package directory name from the repository URL, not the package name.
• Migrate spm_parser to a separate repository.
  • Migrate the code
  • Create a release process that builds binaries for MacOS and Linux
• Update rules_spm to download spm_parser.
• Add code to interesting_deps example that uses libwebp. example
• Add support for packages with Objective-C.
• Add support for packages with C that do have special settings (e.g. CSetting).
• Implement functional tests for spm_parser.
• Enable integration tests that test build_mode set to bazel.

Working Examples

• simple
• simple_with_dev_dir
• simple_revision
• interesting_deps - Need to derive the source package name from the repo URL and not the pkg_name.
• simple_with_binary - The binary that we define works. The swiftformat and swiftlint do not work due to special C settings.
• ios_sim - Requires support for special C settings.
• local_package - Requires support for special C settings.
• public_hdrs - Need to find source path from path setting in the Package.swift. (e.g. TrustKit)
• vapor - Requires support for special C settings.

[cgrindel]

Looks like I will need to parse the Package.swift to get any C-specific settings/options.

Example: swift-nio with CSetting
Example: Yams with CSetting

Example code to parse Package.swift.

[cgrindel]

Notes on getting libwebp in interesting_deps example to build:

• Need to add the following includes to find the private headers.

    includes = [
        "include/webp",
        "libwebp",
        "libwebp/src",
    ],

[cgrindel]

Over the past week or so, I have been working on a proof-of-concept to understand what it would take to introduce a build_mode to spm_repositories. Setting this attribute to bazel would change how rules_spm works underneath the covers. Instead of building the dependent packages using SPM, it would generate Bazel build files allowing the targets to be built using the normal Bazel toolchains. I was able to successfully build and use Swift packages with Swift targets and generic clang targets. Unfortunately, configuring clang targets with custom build settings requires more information than a package description provides.

To be able to properly build clang targets with custom build settings, one needs to retrieve and apply these settings (e.g. defines, public header paths). The current scheme for downloading and reading these manifest files during the repository fetch phase requires code that can parse the manifest and present the information in a means that is readable by Starlark code. Unfortunately, there is no Swift package manifest parser written in Starlark. One can write a parser in Starlark and maintain it as SPM evolves. The question is whether that is the best path forward.

This conundrum is not new to this project. The same decision had to be made when figuring out how to read data from module.modulemap files. At the time, we opted to write a modulemap parser in Starlark thinking that the format was fairly mature and did not appear to be changing much.

To avoid having to write a custom parser, I wrote a simple Swift binary that reads a Swift package manifest and dumps information about it to JSON. The JSON is then read by Starlark code. The approach works. The problem is how one gets and uses the binary from code that is run during Bazel's fetch phase. As of this writing, the current code uses swift run to build and execute the parser. This works OK if there is only one spm_repositories declaration. However, if there are multiple declarations, one will experience an error due to two Swift processes trying to build the same package. I could work around this by ensuring that two processes never build the same code simultaneously. However, that would require duplicating the code and the compilation (ugh) or inventing some kind of semaphore that only allows one build to run (double ugh).

So, I am left with one of a few options:

1. Solve the duplicate build problem for spm_parser. (Yuck)
2. Write a Swift package manifest parser in Starlark. This would allow rules_spm to provide a repository rule that magically does the heavy lifting that it does today.
3. Go completely different direction with rules_spm. Instead of providing a repository rule, it would provide an executable target that generates Bazel build files similar to google/cargo-raze. (Thanks to @keith for making me aware of this project.)

[cgrindel]

Created #157 asking the community how they would like to move forward.