Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support registry fallback chains #35

Merged
merged 11 commits into from
Nov 10, 2021
Merged

Conversation

magnalite
Copy link
Member

@magnalite magnalite commented Oct 12, 2021

This PR adds support for registry fallback chains. See #34

To support sourcing packages from multiple registries each registry can elect a list of other registries which can be looked at as additional sources of packages. The primary use case is to enable us to create registry which contains packages that depend on packages in another registry, for example a private registry with packages depending on the public registry.

This PR includes significant restructuring of the test registry. Previously tests completely ignored the registry listed in the test package manifest files which isn't acceptable to test this sort of use case. Due to this, the tests have been shifted around and manifest files are now respected, even in tests.

As example we can see a new example test package's manifest

[package]
name = "private/private-with-public-dependency"
version = "0.1.0"
license = "MIT"
realm = "server"
registry = "test-registries/tertiary-registry"

[server-dependencies]
Minimal = "biff/minimal@0.1.0"

Where the manifest for biff/minimal contains

[package]
name = "biff/minimal"
version = "0.1.0"
license = "MIT"
realm = "server"
registry = "test-registries/primary-registry"

These belong to two different registries, primary-registry and tertiary-registry. The tertiary test registry contains the following index/config.json.

{
  "api": "http://localhost",
  "fallback_registries": ["../primary-registry"]
}

Tests look at all of these manifests and correctly resolve packages across multiple registries like this. I see this as a significant increase in the robustness of our integration testing. This also works against a real registry using GitHub/hosted api.

This PR:

  • Rearranges the test registry to support multiple test registries
  • Uses test package manifests to direct registry lookup rather than ignoring them and using a de facto test registry
  • Adds a fallback_registries field to the index config.json
  • Implements constructing a package source list by inspecting registry fallbacks
  • Searches through sources in that source list during package resolution until candidates are found
  • Improves error message for when a package candidate cannot be found
  • Indicates which index is being looked at when updating local copies

@magnalite magnalite marked this pull request as ready for review October 13, 2021 17:37
Copy link

@evaera evaera left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments.

We should implement fallback chains for the in-memory registry so we can write more tests. One test I'd like to see is installing, then a registry closer to the front of the chain adds the same package.

Also, I didn't leave comments for the match statements matching on true/false, but we should convert those to if expressions as well.

src/package_index.rs Outdated Show resolved Hide resolved
src/package_source.rs Outdated Show resolved Hide resolved
@@ -108,6 +108,10 @@ impl PackageSource for InMemoryRegistrySource {

Ok(entry.contents.clone())
}

fn fallback_sources(&self) -> anyhow::Result<Vec<PackageSourceId>> {
todo!("Implement in-memory fallback sources");
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use anyhow::bail here? There's no reason to panic since this is already a fallible function

Copy link
Member Author

@magnalite magnalite Nov 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there something wrong with a panic here? I'm not sure I understand why a bail would be preferable to a panic since this is code that shouldn't be running anyway. I like how the todo! here implies that this is something that needs to be implemented rather than some condition that hasn't been met.

src/package_source/registry.rs Outdated Show resolved Hide resolved
src/resolution.rs Outdated Show resolved Hide resolved
src/resolution.rs Outdated Show resolved Hide resolved
@magnalite
Copy link
Member Author

We should implement fallback chains for the in-memory registry so we can write more tests. One test I'd like to see is installing, then a registry closer to the front of the chain adds the same package.

I agree with this however I'm going to add it as a new issue. It's something we can always come back to and I'm eager to get this change finally merged.

@magnalite magnalite merged commit c569a71 into main Nov 10, 2021
@magnalite magnalite deleted the support-registry-fallbacks branch November 10, 2021 17:06
@magnalite magnalite added this to the Essential feature complete milestone Jun 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants