-
Notifications
You must be signed in to change notification settings - Fork 772
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for named and explicit indexes #7481
Conversation
This still needs docs (beyond the inline documentation), but I'd like to align on the semantics described in the PR summary first. |
4561e5d
to
c6d7a92
Compare
// /// structured as a flat list of distributions (e.g., `--find-links`). In both cases, indexes | ||
// /// can point to either local or remote resources. | ||
// #[serde(default)] | ||
// pub r#type: IndexKind, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left this in for now but can remove before merging. Eventually I want this to support --find-links
.
98c909f
to
e6f9d9a
Compare
Yeah this seems wrong. Why is it this way? Do we need an explicit Do we set the explicit tag during a How does index pinning work for transitive dependencies? How can we teach the PyPI fallback behavior? Are there any bad user experiences where we could suggest adding |
Well... we could just invert the priority, if that's what you mean (such that the new index stuff comes last, and the legacy arguments come first). Then the CLI arguments would work as expected. As-is, though, we don't have a way to differentiate between indexes passed on the command line (with
Yeah roughly this.
We don't as of this PR... We can though. I think we probably should? I guess by default we add the index, and make it explicit for that package? The unfortunate thing is that we need a name for the index in that case (as we discussed on Discord). Alternatively, we could just add a
It doesn't have any effect on transitive dependencies. I don't know that it can or should, honestly, because transitive dependencies can be required from multiple different first-party dependencies that could come from different indexes. I think this would be extremely hard to implement correctly and could lead to confusing behavior. |
I prefer to make |
Did you consider allowing pinning packages to indexes using glob-patterns as mentioned here? AFAIK that is the only solution for pinning transitive dependencies that is straightforward enough but still useful for a set of real use-cases. |
There's nothing stopping us from supporting that in addition to the schema described above. It strikes me as somewhat backwards but I understand why it's useful. My only concern is that we're complicating the schema and creating two ways to assign a package to an index. |
@charliermarsh I know it's painful/stupid/messy, but the regex assignment is still really useful for handling transitive dependencies in corporate environments. I don't think there are easy alternatives given the default assumption that python wants to make about indices being equivalent. |
I wonder if we should have a separate schema for pinning transitive packages to a defined index? It could allow globs as well. I'm not sure what all the trade-offs are. |
@zanieb -- That's a good call. Instead of putting this on the index schema, we could have a separate table for assigning packages to named indexes. |
e6f9d9a
to
d404df8
Compare
I'll probably be pushing to this branch a lot over the next few days, so you may want to unsubscribe if the notifications are annoying! |
3208ab0
to
518f3b3
Compare
6b44e69
to
23a6de0
Compare
23a6de0
to
f9ee2a9
Compare
Hey @charliermarsh I wanted to ask what are the steps to get it merged and released and if I can somehow assist you? I tested it locally by installing details on the setup
Thanks in advance 🙏 |
I’m planning to release it early next week. Sorry for the delay — I’m on vacation today, then at EuroRust later in the week. |
Hey @charliermarsh Amazing work, would be amazing if in this PR we could have the That would help my team and me not to add a library to write/dump the data.toml file 🙏🏽 |
|
That's awesome, thank you so much, really excited about these features 😄 |
f1be5d9
to
4b0e753
Compare
4b0e753
to
9e54908
Compare
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.4.22` -> `0.4.24` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.4.24`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0424) [Compare Source](astral-sh/uv@0.4.23...0.4.24) ##### Bug fixes - Fix Python executable name in Windows free-threaded Python distributions ([#​8310](astral-sh/uv#8310)) - Redact index credentials from lockfile sources ([#​8307](astral-sh/uv#8307)) - Respect `UV_INDEX_` rather than `UV_HTTP_BASIC_` as documented ([#​8306](astral-sh/uv#8306)) - Improve sources deserialization errors ([#​8308](astral-sh/uv#8308)) ##### Documentation - Correct pytorch-to-torch reference in docs ([#​8291](astral-sh/uv#8291)) ### [`v0.4.23`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0423) [Compare Source](astral-sh/uv@0.4.22...0.4.23) This release introduces a revamped system for defining package indexes, as an alternative to the existing pip-style `--index-url` and `--extra-index-url` configuration options. You can now define named indexes in your `pyproject.toml` file using the `[[tool.uv.index]]` table: ```toml [[tool.uv.index]] name = "pytorch" url = "https://download.pytorch.org/whl/cpu" ``` Packages can be pinned to a specific index via `tool.uv.sources`, to ensure that a given package is installed from the correct index. For example, to ensure that `torch` is *always* installed from the `pytorch` index: ```toml [tool.uv.sources] torch = { index = "pytorch" } [[tool.uv.index]] name = "pytorch" url = "https://download.pytorch.org/whl/cpu" ``` Indexes can also be marked as `explicit = true` to prevent packages from being installed from that index unless explicitly pinned. For example, to ensure that `torch` is installed from the `pytorch` index, but all other packages are installed from the default index: ```toml [tool.uv.sources] torch = { index = "pytorch" } [[tool.uv.index]] name = "pytorch" url = "https://download.pytorch.org/whl/cpu" explicit = true ``` To define an additional index outside a `pyproject.toml` file, use the `--index` command-line argument (or the `UV_INDEX` environment variable); to replace the default index (PyPI), use the `--default-index` command-line argument (or `UV_DEFAULT_INDEX`). These changes are entirely backwards-compatible with the deprecated `--index-url` and `--extra-index-url` options, which continue to work as before. See the [Index](https://docs.astral.sh/uv/configuration/indexes/) documentation for more. ##### Enhancements - Add index URLs when provided via `uv add --index` or `--default-index` ([#​7746](astral-sh/uv#7746)) - Add support for named and explicit indexes ([#​7481](astral-sh/uv#7481)) - Add templates for popular build backends ([#​7857](astral-sh/uv#7857)) - Allow multiple pinned indexes in `tool.uv.sources` ([#​7769](astral-sh/uv#7769)) - Allow users to incorporate Git tags into dynamic cache keys ([#​8259](astral-sh/uv#8259)) - Pin named indexes in `uv add` ([#​7747](astral-sh/uv#7747)) - Respect named `--index` and `--default-index` values in `tool.uv.sources` ([#​7910](astral-sh/uv#7910)) - Update to latest PubGrub version ([#​8245](astral-sh/uv#8245)) - Enable environment variable authentication for named indexes ([#​7741](astral-sh/uv#7741)) - Avoid showing lower-bound warning outside of explicit lock and sync ([#​8234](astral-sh/uv#8234)) - Improve logging during lock errors ([#​8258](astral-sh/uv#8258)) - Improve styling of `requires-python` warnings ([#​8240](astral-sh/uv#8240)) - Show hint in resolution failure on `Forbidden` (`403`) or `Unauthorized` (`401`) ([#​8264](astral-sh/uv#8264)) - Update to latest `cargo-dist` version (includes new installer features) ([#​8270](astral-sh/uv#8270)) - Warn when patch version in `requires-python` is implicitly `0` ([#​7959](astral-sh/uv#7959)) - Add more context on client errors during range requests ([#​8285](astral-sh/uv#8285)) ##### Bug fixes - Avoid writing duplicate index URLs with `--emit-index-url` ([#​8226](astral-sh/uv#8226)) - Fix error leading to out-of-bound panic in `uv-pep508` ([#​8282](astral-sh/uv#8282)) - Fix managed distributions of free-threaded Python on Windows ([#​8268](astral-sh/uv#8268)) - Fix selection of free-threaded interpreters during default Python discovery ([#​8239](astral-sh/uv#8239)) - Ignore sources in build requirements for non-source trees ([#​8235](astral-sh/uv#8235)) - Invalid cache when adding lower bound to lockfile ([#​8230](astral-sh/uv#8230)) - Respect index priority when storing credentials ([#​8256](astral-sh/uv#8256)) - Respect relative paths in `uv build` sources ([#​8237](astral-sh/uv#8237)) - Narrow what the pip3.<minor> logic drops from entry points. ([#​8273](astral-sh/uv#8273)) ##### Documentation - Add some additional notes to `--index-url` docs ([#​8267](astral-sh/uv#8267)) - Add upgrade note to README ([#​7937](astral-sh/uv#7937)) - Remove note that "only a single source may be defined for each dependency" ([#​8243](astral-sh/uv#8243)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
does this support, or is support planned for, using API keys for authentication against private registries (ie artifactory) rather than using username/password? |
You can use an API key in the PASSWORD variable — it works for anything that supports HTTP Basic Authentication. |
thanks for the support! i'm trying this now and seeing issues. For context, i have a project which has one dependency from a private artifactory registry. with or without credentials exported to the environment i see the following output:
the dependencies are specified as follows: dependencies = [
"my-private-package~=2.0.0",
# ...
]
[tool.uv.sources]
my-private-package = {index = "mycompany-py-release"}
[[tool.uv.index]]
name = "artifactory-pypi"
url = "https://artifactory.mycompany.com/artifactory/api/pypi/pypi/simple"
default = true
[[tool.uv.index]]
# Requires:
name = "mycompany-py-release"
url = "https://artifactory.mycompany.com/artifactory/api/pypi/my-private-package/simple"
explicit = true with export UV_INDEX_MYCOMPANY_PY_RELEASE_USERNAME={username}
export UV_INDEX_MYCOMPANY_PY_RELEASE_PASSWORD={artifactory token} |
You should open a new issue for this and we can discuss there |
Any news about this feature? This is critical for some users |
Summary
This PR adds a first-class API for defining registry indexes, beyond our existing
--index-url
and--extra-index-url
setup.Specifically, you now define indexes like so in a
uv.toml
orpyproject.toml
file:You can also provide indexes via
--index
andUV_INDEX
, and override the default index with--default-index
andUV_DEFAULT_INDEX
.Index priority
Indexes are prioritized in the order in which they're defined, such that the first-defined index has highest priority.
Indexes are also inherited from parent configuration (e.g., the user-level
uv.toml
), but are placed after any indexes in the current project, matching our semantics for other array-based configuration values.You can mix
--index
and--default-index
with the legacy--index-url
and--extra-index-url
settings; the latter two are merely treated as unnamed[[tool.uv.index]]
entries.Index pinning
If an index includes a name (which is optional), it can then be referenced via
tool.uv.sources
:If an index is marked as
explicit = true
, it can only be used via such references, and will never be searched implicitly:Indexes defined outside of the current project (e.g., in the user-level
uv.toml
) can not be explicitly selected.(As of now, we only support using a single index for a given
tool.uv.sources
definition.)Default index
By default, we include PyPI as the default index. This remains true even if the user defines a
[[tool.uv.index]]
-- PyPI is still used as a fallback. You can mark an index asdefault = true
to (1) disable the use of PyPI, and (2) bump it to the bottom of the prioritized list, such that it's used only if a package does not exist on a prior index:Name reuse
If a name is reused, the higher-priority index with that name is used, while the lower-priority indexes are ignored entirely.
For example, given:
The
https://test.pypi.org/simple
index would be ignored entirely, since it's lower-priority thanhttps://download.pytorch.org/whl/cu121
but shares the same name.Closes #171.
Future work
uv add
should automatically write--index
entries to thepyproject.toml
file.