Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue: Stabilize uv.lock format #4893

Closed
6 tasks done
Tracked by #3347
konstin opened this issue Jul 8, 2024 · 6 comments
Closed
6 tasks done
Tracked by #3347

Tracking issue: Stabilize uv.lock format #4893

konstin opened this issue Jul 8, 2024 · 6 comments
Assignees
Labels
preview Experimental behavior

Comments

@konstin
Copy link
Member

konstin commented Jul 8, 2024

This is a tracking issue for making breaking changes the schema and contents of uv.lock before the release.

  • Remove extra duplication from lockfile #4888
  • Sort dependencies before sdist in lockfile #4889
  • Move the resolution options (resolution lowest, exclude newer, prerelease mode) under a newer header resolver-options to group them and make them distinct from the rest of the top level entries.
  • Rename distribution to package or packages. This is not a distribution entry, the distributions are the source dists and wheels. package is also a more familiar name and closer to PEP 751, packages would be the same as PEP 751.
  • If only one index is used, move the source = { registry = "https://pypi.org/simple" } to the top of the file and out of the [[distribution]]. This shrinks the file a bit and removed the duplication.
  • Try to make the lockfile more concise. Could we e.g. remove redundant source line? The lockfile is long and should try to get it as small as possible without sacrificing readability and diffability.
@konstin konstin added the preview Experimental behavior label Jul 8, 2024
@konstin konstin changed the title Stabilize lockfile format Stabilize uv.lock format Jul 8, 2024
@T-256
Copy link
Contributor

T-256 commented Jul 8, 2024

  • Try to make the lockfile more concise. Could we e.g. remove redundant source line?

it could use source at top-level which implies that distributions without source are using it:

version = 1
source = { registry = "https://pypi.org/simple" }
# or:
# source.registry = "https://pypi.org/simple"

[[distribution]]
name = "a"
version = "0.1.0"
sdist = { url = "https://example.com", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 0 }

[[distribution]]
name = "a"
version = "0.1.1"
sdist = { url = "https://example.com", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 0 }

[[distribution]]
name = "b"
version = "0.1.0"
source = { editable = "path/to/dir" }
sdist = { url = "https://example.com", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 0 }

Idk, but I expected checksum instead of hash. also it used by cargo.lock

  • Rename sdist to source-dist.

I vote to don't change it, or source-distribution for consistency, or source-code/code/code-distribution.

@chrisrodrigue
Copy link

@T-256

This is nitpicking semantics but I think hash and checksum are often used interchangeably. I couldn't find a terrible amount of info on it but I did find this glossary definition suggesting hash is the more correct term. I think checksum reflects the intended use case of the hash, but the hash can theoretically be used as more than just a checksum.

@konstin konstin moved this to Backlog in uv: release-ready Jul 9, 2024
@konstin konstin changed the title Stabilize uv.lock format Tracking issue: Stabilize uv.lock format Jul 9, 2024
@tinovyatkin
Copy link

Can we have an option to exclude source completely? It may include per user specific credentials in case of private registries. Something similar to NPM omit-lockfile-registry-resolved option?

@zanieb
Copy link
Member

zanieb commented Aug 5, 2024

@tinovyatkin we're planning on redacting credentials from the lockfile (see #5119), is that going to be sufficient or is there more to your use-case?

@tinovyatkin
Copy link

tinovyatkin commented Aug 5, 2024

@tinovyatkin we're planning on redacting credentials from the lockfile (see #5119), is that going to be sufficient or is there more to your use-case?

Unfortunately in our case we are using "authentication proxies" on developers machines (so address starts like http://127.0.0.1....) and some specific address on CI builds.

Can source at top level contain the original source where the address was obtained? Like "source": "config" or "source": "environment"?

konstin added a commit that referenced this issue Aug 7, 2024
Currently, we're repeating the same `source` line for every package, so the lockfiles have a lot of:

```toml
source = { registry = "https://pypi.org/simple" }
```

This PR introduces a top level `default-source` entry set to the default index URL, if any. When the source matches, we don't repeat the `source` entry. This reduces the number of lines in `uv.lock` noticeably across the board:

* A small data science project: 421 -> 394
* A small bot: 455 -> 426
* Transformers: 5683 -> 5419
* Warehouse: 4632 -> 4306
* Airflow: 2709 -> 2576

Caveat: We don't have good multi-index coverage (#5882).

3/3 for #4893
charliermarsh pushed a commit that referenced this issue Aug 7, 2024
There are three options that determine resolver behavior:

* resolution mode
* prerelease mode
* exclude newer

They are different from the other top level options: If they mismatch,
we recreate the resolution. To distinguish them from the rest of the
lockfile, we group them under an `[options]` header.

1/3 for #4893
konstin added a commit that referenced this issue Aug 7, 2024
Currently, we're repeating the same `source` line for every package, so the lockfiles have a lot of:

```toml
source = { registry = "https://pypi.org/simple" }
```

This PR introduces a top level `default-source` entry set to the default index URL, if any. When the source matches, we don't repeat the `source` entry. This reduces the number of lines in `uv.lock` noticeably across the board:

* A small data science project: 421 -> 394
* A small bot: 455 -> 426
* Transformers: 5683 -> 5419
* Warehouse: 4632 -> 4306
* Airflow: 2709 -> 2576

Caveat: We don't have good multi-index coverage (#5882).

3/3 for #4893
konstin added a commit that referenced this issue Aug 7, 2024
Currently, we're repeating the same `source` line for every package, so the lockfiles have a lot of:

```toml
source = { registry = "https://pypi.org/simple" }
```

This PR introduces a top level `default-source` entry set to the default index URL, if any. When the source matches, we don't repeat the `source` entry. This reduces the number of lines in `uv.lock` noticeably across the board:

* A small data science project: 421 -> 394
* A small bot: 455 -> 426
* Transformers: 5683 -> 5419
* Warehouse: 4632 -> 4306
* Airflow: 2709 -> 2576

Caveat: We don't have good multi-index coverage (#5882).

3/3 for #4893
konstin added a commit that referenced this issue Aug 8, 2024
Currently, we're repeating the same `source` line for every package, so the lockfiles have a lot of:

```toml
source = { registry = "https://pypi.org/simple" }
```

This PR introduces a top level `default-source` entry set to the default index URL, if any. When the source matches, we don't repeat the `source` entry. This reduces the number of lines in `uv.lock` noticeably across the board:

* A small data science project: 421 -> 394
* A small bot: 455 -> 426
* Transformers: 5683 -> 5419
* Warehouse: 4632 -> 4306
* Airflow: 2709 -> 2576

Caveat: We don't have good multi-index coverage (#5882).

3/3 for #4893
@konstin konstin moved this from Backlog to In progress in uv: release-ready Aug 8, 2024
charliermarsh added a commit that referenced this issue Aug 8, 2024
Currently, the entry for a package+version+source table is called
`distribution`. That is incorrect, the `sdist` and `wheel` fields inside
of that table are distributions, the table itself is for a package. We
also align ourselves closer with PEP 751.

I went through `lock.rs` and renamed all occurrences of "distribution"
that actually referred to a "package".

This change invalidates all existing lockfiles.

Bikeshedding: Do we call it `package` or `packages`? See also
python/peps#3877

`package` is nice because it looks like a header:

```toml
[[package]]
name = "anyio"
version = "4.3.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
    { name = "idna" },
    { name = "sniffio" },
]
sdist = { url = "https://files.pythonhosted.org/packages/db/4d/3970183622f0330d3c23d9b8a5f52e365e50381fd484d08e3285104333d3/anyio-4.3.0.tar.gz", hash = "sha256:f75253795a87df48568485fd18cdd2a3fa5c4f7c5be8e5e36637733fce06fed6", size = 159642 }
wheels = [
    { url = "https://files.pythonhosted.org/packages/14/fd/2f20c40b45e4fb4324834aea24bd4afdf1143390242c0b33774da0e2e34f/anyio-4.3.0-py3-none-any.whl", hash = "sha256:048e05d0f6caeed70d731f3db756d35dcc1f35747c8c403364a8332c630441b8", size = 85584 },
]
```

`packages` is nice because the field is not a single entry, but a list.

2/3 for #4893

---------

Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
@konstin
Copy link
Member Author

konstin commented Aug 9, 2024

We have the format ready for the release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
preview Experimental behavior
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

5 participants