Issue integrating Github SCM Branch with Registry Package

[jonnyklemmer]

Is it reproducible with SwiftPM command-line tools: swift build, swift test, swift package etc?

• Confirmed reproduction steps with SwiftPM CLI.

Description

Initially want to caveat that this issue may not be a bug - and instead a limitation of SPM/Registries or a misunderstanding of the tooling. Hopefully by describing the issue I can get some clarity to which it is. Thanks in advance for the support!

When setting up a project which consists of packages resolved via Swift Package Registry our desire is to also support temporary development via Github branches. The goal here being that tagged and shipped releases can utilize a semver versioned package via the registry, but developers working on local iterations of code could easily point to a branch of the repository. The following is the issues I have come across with our current approach, but if theres better/alternate ways to "solve" this workflow I'm all ears!

The following Package files illustrate the issue with a top level "root project" and a child "feature package" - who both share the same dependency "Pillars" (brought in via registries and SCM).

Sample Package Files:

Root Project - Package.swift

let package = Package(
    name: "RootProject",
    products: [
        .library(
            name: "RootProject",
            targets: ["RootProject"]),
    ],
    dependencies: [
        .package(name: "FeaturePackage", path: "./Packages/FeaturePackage"),
        // Github Package
        .package(url: "https://github.com/company-name/mpe.foundation.pillars.git", branch: "testbranch")
    ],
    targets: [
        .target(
            name: "RootProject",
            dependencies: [
                "FeaturePackage",
                .product(name: "Pillars", package: "mpe.foundation.pillars")
            ]
        ),
    ]
)

Child Feature - Package.swift

let package = Package(
    name: "FeaturePackage",
    products: [
        .library(
            name: "FeaturePackage",
            targets: ["FeaturePackage"]),
    ],
    dependencies: [
		// Registry Package (via Artifactory)
        .package(id: "mpe.Pillars", exact: "4.2.0")
    ],
    targets: [
        .target(
            name: "FeaturePackage",
            dependencies: [
                .product(name: "Pillars", package: "mpe.Pillars")
            ]
        ),
    ]
)

Expected behavior

Ideally the root project's package manifest would supersede the children and it would resolve the package via the github branch. The transitive dependency of the child feature package would therefore also be fulfilled by the github branch package.

Actual behavior

It seems given the two variations come from SCM & a Package Registry (and the Github repo last path component doesn't match the registry scope.name) that it is indeterministic which package is resolved.

Initially this was seen via this error:

error: multiple similar targets 'Pillars' appear in registry package 'mpe.Pillars' and source control package 'mpe.foundation.pillars', this may indicate that the two packages are the same and can be de-duplicated by activating the automatic source-control to registry replacement, or by using mirrors. if they are not duplicate consider using the `moduleAliases` parameter in manifest to provide unique names

NOTE: It is unclear how the flags --use-registry-identity-for-scm or --replace-scm-with-registry can be used to solve this issue (assuming thats what "de-duplicated by activating the automatic source-control to registry replacement" refers to). Additionally with mirrors it isn't clear how a mirror can be used to map between a package registry and a git URL.

Currently the swift package resolve command is now simply successful - but pulls in the package from the registry and ignores the branch specified by the root project manifest.

Steps to reproduce

Configure a package with the attached package files, a 3rd "shared" dependency package (in this case "Pillars"), and host it both on github and a package registry.

Apologies this isn't fully available as an open-source reproduction environment given it utilizes private frameworks and an Artifactory package registry - if thats a blocker let me know and can work to provide easier reproduction.

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.9.0

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.87.3 Apple Swift version 5.9.2 (swiftlang-5.9.2.2.56 clang-1500.1.0.2.5)
Target: arm64-apple-macosx14.0
Darwin [redacted] 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:44 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6000 arm64u

[bnbarham]

There's two options here I believe:

1. Always use the registry dependency and then use mirrors - note that set-mirror just takes arbitrary strings and not necessarily URLs and thus eg. swift package config set-mirror --original mpe.Pillars --mirror https://github.com/company-name/mpe.foundation.pillars.git should work for your development purposes IIUC.
2. Make sure that the repository URL is specified in the registry for that package (https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#45-lookup-package-identifiers-registered-for-a-url), use the SCM URL all the time, and then use --replace-scm-with-registry to override it with the registry. I suspect (1) is more what you want though.

[jonnyklemmer]

There's two options here I believe:

1. Always use the registry dependency and then use mirrors - note that set-mirror just takes arbitrary strings and not necessarily URLs and thus eg. swift package config set-mirror --original mpe.Pillars --mirror https://github.com/company-name/mpe.foundation.pillars.git should work for your development purposes IIUC.
2. Make sure that the repository URL is specified in the registry for that package (https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#45-lookup-package-identifiers-registered-for-a-url), use the SCM URL all the time, and then use --replace-scm-with-registry to override it with the registry. I suspect (1) is more what you want though.

Appreciate the reply @bnbarham!

I do agree option 1 is closest to what we want, however effectively we need to both change the source from a registry -> GitHub, but also to switch from a concrete registry tag (ex .package(id: "mpe.Pillars", exact: "4.2.0") to a developers branch (ex .package(url: "https://github.com/company-name/mpe.foundation.pillars.git", branch: "testbranch")). It seems with the mirror approach we can only mirror the location it pulls from, but that would still require the same matching tag to exist.

The use case is to support primarily using versioned registry packages where as-needed when working at an app level you have the flexibility to test changes to a dependency on a branch across the entire dependency graph. For comparison in a Cocoapods environment as long as the semver requirements of "FeaturePackage" podspec were met by the "RootProject" pod file (pointed to a branch) the dependency manager would resolve the dependency from the branch across all child frameworks.

From memory of my earlier investigations into --replace-scm-with-registry I agree I believe that setup isn't what we want, but let me know based on the updates above if there's a better path down that route.

[bnbarham]

Appreciate the reply @bnbarham!

Sorry for the delay!

It seems with the mirror approach we can only mirror the location it pulls from, but that would still require the same matching tag to exist.

If you specify a local path it would use that instead, regardless of any version specified. Would that work for you here? It would require checking out all your repositories locally first, but is probably the best workaround we have right now.

FWIW I don't think this issue is really about the package registry or not - you'd have the same problem if you just wanted to override a regular package URL with another version. So we could probably broaden this issue to track overriding in general.

[Robin-Kunde-ck]

@jonnyklemmer Which tools version did you specify in the Package.swift files?

[jonnyklemmer]

@jonnyklemmer Which tools version did you specify in the Package.swift files?

Sorry for the delayed response but this was all with 5.9

[Robin-Kunde-ck]

@jonnyklemmer I'm wondering if #7912 is part of the problem in your case.