Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Private Projects #8214

Open
Tracked by #238 ...
johnthagen opened this issue Oct 15, 2024 · 12 comments
Open
Tracked by #238 ...

Private Projects #8214

johnthagen opened this issue Oct 15, 2024 · 12 comments
Labels
documentation Improvements or additions to documentation

Comments

@johnthagen
Copy link

johnthagen commented Oct 15, 2024

Is it possible to mark a uv project as private such that uv publish is completely disabled? For private code this is a desirable feature.

Something like:

[tool.uv]
private = true

NPM has a similar "private": true:

If you set "private": true in your package.json, then npm will refuse to publish it.

This is a way to prevent accidental publication of private repositories.

In Cargo, this is controlled by the publish flag.

To prevent a package from being published to a registry (like crates.io) by mistake, for instance to keep a package private in a company, you can omit the version field. If you’d like to be more explicit, you can disable publishing:

[package]
# ...
publish = false

For some additional motivation, see a similar feature request for Poetry

@zanieb
Copy link
Member

zanieb commented Oct 15, 2024

This exists already as a Private :: Do Not Upload classifier

@zanieb
Copy link
Member

zanieb commented Oct 15, 2024

This is mentioned over in python-poetry/poetry#1537 (comment) — we might want to include a private mode that avoids attempting to upload to the registry at all.

cc @konstin

@zanieb zanieb added enhancement New feature or improvement to existing functionality needs-decision Undecided if this should be done labels Oct 15, 2024
@johnthagen
Copy link
Author

@zanieb Yeah, I personally feel like the Private :: Do Not Upload classifier is not 100% ideal and having something built in at the package manager level like NPM or Cargo would be more reliable (and also probably easier to teach).

@konstin
Copy link
Member

konstin commented Oct 15, 2024

I'd love to support something like tool.uv.private = true. The only blocker for this is that currently, uv publish is independent of pyproject.toml: We will upload the files you hand uv publish and if you hand it a wheel, there's no pyproject.toml anymore. We also aren't allowed to set "Private :: Do Not Upload" for you when tool.uv.private = true is set, since we have to respect project.classifiers.

For now, we should start hinting users towards the Private :: Do Not Upload classifier in the docs. It's not the most elegant solution, but it's a reliable way to prevent accidental publishing.

@zanieb
Copy link
Member

zanieb commented Oct 19, 2024

The only blocker for this is that currently, uv publish is independent of pyproject.toml: We will upload the files you hand uv publish and if you hand it a wheel, there's no pyproject.toml anymore.

I understand that in some contexts we may not know where the wheels come from, but I feel like we should still be able to read project metadata and configuration when publishing artifacts for builds in a project. This seems like a fairly important base capability for uv publish.

We also aren't allowed to set "Private :: Do Not Upload" for you when tool.uv.private = true is set, since we have to respect project.classifiers.

How "not allowed" is this? Couldn't we append this when building the wheel and source dists?

@mahyarmirrashed
Copy link

I agree with @zanieb on this. It's fairly important to protect users against shooting themselves in the foot when they accidentally attempt to publish something that is marked as private.
As a user, I can provide the following perspective/expected workflow.

  1. If a project is created with uv init, add the private = true by default. The vast majority of projects are not public. Being proactive here brings less harm than the contrary;
  2. If a project has the "Private :: Do Not Upload" classifier, warn the user that it must be removed in order to continue with publishing;
  3. If a project is marked with private = true and the "Private :: Do Not Upload" classifier is not specified, warn the user that they should add it, link to https://pypi.org/classifiers/, and do not publish;
  4. If a project is marked with private = true and the "Private :: Do Not Upload" classifier is specified, warn the user that it is set and therefore, do not publish.

This respects the apparent rule of not setting the classifier automatically while still protecting the user as much as possible. I strongly encourage not adding any --override parameters either. Those usually are a bad idea for users that blindly use them without understanding the consequences and reasons for their errors.

@konstin
Copy link
Member

konstin commented Oct 19, 2024

There's two options to implementing private project: METADATA and pyproject.toml.

Metadata

One option is going through the METADATA file and making sure "Private :: Do Not Upload" ends up in the source dist and wheel.

The non-uv solution is adding classifiers = ["Private :: Do Not Upload"] to your pyproject.toml:

[project]
name = "foo-internal"
version = "0.1.0"
classifiers = ["Private :: Do Not Upload"]

This prevents publishing to PyPI.

(There is a separate question that alternative registries such as GitLab have options to host packages both as public and private, but I'm skipping over that here.)

How "not allowed" is this? Couldn't we append this when building the wheel and source dists?

PEP 621 is unambiguous that we must not append to static parts of [project] when building the wheel and source dists, this is required for the ability to read pyproject.toml without a build. There is an escape hatch that requires explicitly declaring specifiers as dynamic, we set the actual classifier on build depending on the value of tool.uv.private, i.e., if we see tool.uv.private = true we transform the metadata on build as if we had classifiers = ["Private :: Do Not Upload"]:

[project]
name = "foo-internal"
version = "0.1.0"
dynamic = ["classifier"]

[tool.uv]
private = true

Through dynamic, we are allowed to emit the following as METADATA:

Metadata-Version: 2.3
Name: foo-internal
Version: 0.1.0
Version: Private :: Do Not Upload

I prefer classifiers = ["Private :: Do Not Upload"] over the above for being self-documenting and more concise.

pyproject.toml

The other option is requiring pyproject.toml to be present.

To check for tool.uv.private, we need to require the pyproject.toml of the (forbidden to be) published packages to be present for all uv publish invocations, for workspace packages this means having all workspace pyproject.toml. If we were to only check if the respective pyproject.toml is present, we would risk ignoring a tool.uv.private.

The other, bigger downside is that this will be ignored when using a standard workflow such as python -m build && twine upload dist/*: Twine, @pypa/gh-action-pypi-publish and other tools will publish this package to pypi with no regards to custom uv configuration, while the classifier protects no matter the tooling mix.

There are other advantages too for requiring pyproject.toml for publish, e.g., reading configuration from it. In the current trade-off, that doesn't tip the scales for me towards switching to a poetry-like uv publish, given that the currently implemented solution is a universally supported self-documenting one-liner:

[project]
name = "foo-internal"
version = "0.1.0"
classifiers = ["Private :: Do Not Upload"]

I agree that cargo and npm have a clearly better state, and would favor the addition of project.publish or project.private to the pyproject.toml spec.

@johnthagen
Copy link
Author

The other option is requiring pyproject.toml to be present.

Is there a third option where if pyproject.toml is present, then uv publish inspects it and if private = true is there, it prevents publishing? So normal uv projects will respect it, but if you're using uv publish without a pyproject.toml, then it's back on you to set the classifiers and it works more like twine?

@zanieb
Copy link
Member

zanieb commented Oct 20, 2024

Yeah I agree, I don't think we need to require it. We could always add a flag that requires we can find it; for people that are concerned about accidentally ignoring their configuration.

konstin added a commit that referenced this issue Nov 3, 2024
Users are concerned that they might accidentally publish their private package to PyPI, thereby exposing company internals and their private source code.

However, for this to happen, the user needs to call `uv publish` without an index option, while having credentials for PyPI set either in environment variables (and the private index credentials not set to those env vars) or in the keyring (without scoping to a package), and that token having the project in scope.

As easiest protection, we recommend **never generating a PyPI token scoped to all projects**, only generating one that is matched to the specific, public project: Without a matching token, no accidental publishing can happen.

---

The `Private :: Do Not Upload` prevent packages from being uploaded to PyPI (https://pypi.org/classifiers/). This is unergonomic: The classifiers are a system for metadata, not for configuration, they are not part of uv's docs and it doesn't have the discoverability of regular settings.

The most evident idea for solving this is `tool.uv.publish`:

```
[tool.uv]
private = true
```

Unfortunately, this doesn't actually offer reliable protection: It only works when the pyproject.toml is present (we currently don't require checkouts for `uv publish`) and it doesn't work with twine or any other publishing tool.

The second idea is translating this to `classifiers = ["Private :: Do Not Upload"]`. Unfortunately, PEP 621 forbids this, it requires that we must not append to static parts of `[project]` when building the wheel and source dists. This is needed for the ability to read `pyproject.toml` without a build. There is an escape hatch that requires explicitly declaring specifiers as dynamic, we set the actual classifier on build depending on the value of `tool.uv.private`, i.e., if we see `tool.uv.private = true` we transform the metadata on build as if we had `classifiers = ["Private :: Do Not Upload"]`:

```toml
[project]
name = "foo-internal"
version = "0.1.0"
dynamic = ["classifier"]

[tool.uv]
private = true
```

Through `dynamic`, we are allowed to emit the following as METADATA:

```text
Metadata-Version: 2.3
Name: foo-internal
Version: 0.1.0
Version: Private :: Do Not Upload
```

Just setting `classifiers = ["Private :: Do Not Upload"]` is, unlike the above, self-documenting and more concise.

---

Given all that we document that you shouldn't create unscoped tokens and that you can use `classifiers = ["Private :: Do Not Upload"]`.

Note that we cannot handle e.g. a GitLab repository that is set to public accidentally. It falls into the domain of registry vendors to have guardrails (e.g. private-by-default registries when the status of the source is unknown)

Fixed #8214
@konstin konstin added documentation Improvements or additions to documentation and removed needs-decision Undecided if this should be done enhancement New feature or improvement to existing functionality labels Nov 3, 2024
@ncoghlan
Copy link

ncoghlan commented Nov 5, 2024

I know packaging tool authors are generally against putting UX details into PEPs, but maybe this is a case where it would make sense to propose an optional private = true flag in the project table that fails the build if classifiers = ["Private :: Do Not Upload"] is missing from the classifier list? (@konstin considered a flag above, but in the context of reading it at publication time, rather than using it as a build time cross check on the classifier list)

That way as long as the project developers are running their CI builds with a version that respects the flag, publication tools will still see the classifier in the metadata.

@johnthagen
Copy link
Author

johnthagen commented Nov 6, 2024

an optional private = true flag in the project table that fails the build if classifiers = ["Private :: Do Not Upload"] is missing from the classifier list?

That is an interesting idea. The downsides I see to this are:

  • Simplicity for the user. Now we need to teach users that ideally they need to state their intention that a project is private twice
  • The "Private :: Do Not Upload" still seems like a non-ideal solution in the first place because it's only after your code is shipped through an HTTPS connection over the network somewhere that then it's blocked from being saved. I get that this is a bit pedantic, but in the presence of SSL proxies, etc it's possible that your code is revealed somewhere
    • You then also need to trust that whatever package repository you have configured (since it could be something other than PyPI like Artifactory - again in the very slim edge case) has properly implemented "Private :: Do Not Upload" and rejects it.
  • What I like about private = true is that (just like NPM and Cargo) it can prevent any transmission of the code from package managers that understand it

But I do like the overall idea of proposing that private = true be in a real PEP eventually. That could resolve some of the concerns that "well Twine will upload the .whl anyway". As Twine, Poetry, etc could also be updated to look for this common field in the current directory (or, bikeshedding, could look upwards for a project.toml in upper directories too) and then all refuse to upload.

@ncoghlan
Copy link

ncoghlan commented Nov 7, 2024

What I like about private = true is that (just like NPM and Cargo) it can prevent any transmission of the code from package managers that understand it

Yeah, that's one of the two main advantages I see in defining project.private formally. The other advantage is this:

That could resolve some of the concerns that "well Twine will upload the .whl anyway".

If the invalid classifier is explicitly present in the metadata, even old versions of Twine (or other tools) won't successfully upload the package.

So even though people (or their dev tools) would need to express "this is private" twice, they'd be doing it for a specific backwards compatibility reason. (and if uv init accepted a --private flag, the dev themselves would still only be marking the package as private once)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

5 participants