Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New python manager for PEP621 #10187

Closed
wwuck opened this issue May 27, 2021 · 24 comments · Fixed by #22082
Closed

New python manager for PEP621 #10187

wwuck opened this issue May 27, 2021 · 24 comments · Fixed by #22082
Assignees
Labels
auto:reproduction A minimal reproduction is necessary to proceed new package manager New package manager support priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others status:in-progress Someone is working on implementation type:feature Feature (new functionality)

Comments

@wwuck
Copy link
Contributor

wwuck commented May 27, 2021

What would you like Renovate to be able to do?

Renovate has existing managers for various python sources. The python community has recently accepted PEP621 as a standard format for providing python package metadata in a pyproject.toml file so that multiple packaging-related tools can consume this data from one location instead of various tool-specific configurations. This metadata includes dependencies/optional-dependencies.

I would like to request that Renovate gains a new manager for PEP621/pyproject.toml to maintain and keep updated the dependencies and optional_dependencies fields.

@wwuck wwuck added priority-5-triage status:requirements Full requirements are not yet known, so implementation should not be started type:feature Feature (new functionality) labels May 27, 2021
@rarkins rarkins added the auto:reproduction A minimal reproduction is necessary to proceed label May 27, 2021
@github-actions
Copy link
Contributor

Hi there,

Help us by making a minimal reproduction repository.

Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction to understand what is needed.

We may close the issue if you (or someone else) have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

@rarkins rarkins added new package manager New package manager support priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others and removed priority-5-triage labels May 27, 2021
@wwuck
Copy link
Contributor Author

wwuck commented May 27, 2021

New package manager questionnaire

Did you read our documentation on adding a package manager?

Basics

Name of package manager

PEP621

What language does this support?

Python

How popular is this package manager?

Relatively new but gaining popularity among the popular python build tools. PEP621 has already been implemented by flit, pdm, trampolim, whey, enscons. PEP621 is also planned for inclusion in setuptools and is under consideration for poetry as well.

Does this language have other (competing?) package managers?

  • Yes (give names)
  • No

PEP621 is not a "package manager" itself, but a common format for defining static metadata that other python build tools/package managers can consume.
E.g. if PEP621 is implemented in setuptools it would be "competing with" and most likely superseding setup.py and setup.cfg for specifying the core python project metadata (which includes the package dependency data that Renovate would care about).


Package File Detection

What type of package files and names does it use?

pyproject.toml

What fileMatch pattern(s) should be used?

This would be similar to the existing poetry package manager in Renovate as they both use pyproject.toml, but PEP621 doesn't use a lock file such as poetry.lock.

Is it likely that many users would need to extend this pattern for custom file names?

  • Yes
  • No

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

No. There is only one file, pyproject.toml, associated with PEP621.


Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

No, there is only a single pyproject.toml file.

Is there a reason why package files need to be parsed together (in serial) instead of independently?

What format/syntax is the package file in?

  • JSON
  • TOML
  • YAML
  • Custom (explain below)

How do you suggest parsing the file?

  • Off the shelf parser
  • Using regex
  • Custom-parsed line by line
  • Other

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, dev dependencies, etc?

  • Yes, required dependencies and optional dependencies
  • No, all dependencies are treated the same

List all the sources/syntaxes of dependencies that can be extracted

https://www.python.org/dev/peps/pep-0621/#dependencies-optional-dependencies
https://www.python.org/dev/peps/pep-0621/#example

Describe which types of dependencies above are supported and which will be implemented in future


Versioning

What versioning scheme does the package file(s) use?

Dependencies are specified according to PEP-508, which itself says that dependency versions are specified according to PEP440.

Does this versioning scheme support range constraints, e.g. ^1.0.0 or 1.x?

  • Supports range constraints (e.g ^1.0.0 or 1.x)
  • No

Is this package manager used for applications, libraries, or both? If both, is there a way to tell which is which?

  • Applications
  • Libraries
  • Both (explain how to tell which is which)

As I understand it, PEP621 is not concerned with application vs library. That would be the job of the build tool/package manager that is consuming the PEP621 metadata.

If ranges are supported, are there any cases when Renovate should pin ranges to exact versions if rangeStrategy=auto?


Lookup

Is a new datasource required? Provide details

  • Yes, provide details.
  • No.

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

Yes, this would be similar to other python package managers implemented by Renovate such as setup.py and setup.cfg. This information is not considered part of the PEP621 spec.

Do the package files contain any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc) that should be used in the lookup procedure?

Yes. Only python3 would be supported as python2 has been EOL as of 2020-01-01.

Will users need the ability to configure language or other constraints using Renovate config?


Artifacts

Are lock files or checksum files used? Are they mandatory?

I think this is be outside the scope of PEP621, in the domain of the build tools/package managers that consume PEP621 metadata. I guess package versions would be specified directly in the relevant section of pyproject.toml?

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or env? Do you recommend the cache be kept or disabled/ignored?

I think this is be outside the scope of PEP621, in the domain of the build tools/package managers that consume PEP621 metadata. E.g. pip, poetry, pipenv.

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

Other

Is there anything else to know about this package manager?

@HonkingGoose
Copy link
Collaborator

@wwuck Thank you for using the issue template, and filling out the questionnaire! 😄

@HonkingGoose
Copy link
Collaborator

@rarkins It seems we still need a minimal reproduction repository? What do you need to proceed?

@rarkins
Copy link
Collaborator

rarkins commented Jun 20, 2021

Yes a repo or more with example use cases would help.

Also, once different tools implement lock files then we may need to rethink how to handle it.

@pradyunsg
Copy link

Here's some repositories with PEP 621 style dependency declarations:

Notice that all of these declarations are "abstract" (i.e. not exact versions/artifacts, but rather a description of what version ranges are compatible) and not "concrete" (like a lock file that pins all artifacts and hashes would be).

@renovatebot renovatebot deleted a comment from github-actions bot May 13, 2022
nuztalgia added a commit to nuztalgia/qibot that referenced this issue Jul 10, 2022
This is purely because Renovate currently does not support `pyproject.toml` as a Python dependency manager: renovatebot/renovate#10187
@nuztalgia
Copy link

nuztalgia commented Jul 10, 2022

Adding a +1 to this request, as I just discovered pyproject.toml wasn't supported by Renovate after migrating to it in one of my repos. 😅 Based on the comments/tags in this issue, it looks like more examples/repros are necessary before it can be considered, so here's some info about my specific user experience with searching for a modern Python dependency manager that's supported by Renovate. I've included snapshots of the relevant dependency specification files and log excerpts from the corresponding Renovate jobs for each manager that I tried out.


Pipfile

The old faithful, and happily supported by Renovate. This is where I originally put my dependencies, before deciding that I wanted to "modernize" my project.

https://github.com/nuztalgia/qibot/blob/566d9c91b53f79b75a135866df40aa21dce56ebe/Pipfile#L6-L10

Log excerpts from job ID #730618240 (click to expand)
DEBUG: latest repository commit
{
  "latestCommit": {
    "hash": "566d9c91b53f79b75a135866df40aa21dce56ebe",
[...]
DEBUG: Matched 1 file(s) for manager pipenv: Pipfile
DEBUG: pre-commit is disabled
DEBUG: pipenv.extractPackageFile()
DEBUG: Found pipenv package files
DEBUG: Found 1 package file(s)
INFO: Dependency extraction complete
{
  "baseBranch": "main",
  "stats": {
    "managers": {
      "pipenv": {
        "fileCount": 1,
        "depCount": 4
      }
    },
    "total": {
      "fileCount": 1,
      "depCount": 4
    }
  }
}

pyproject.toml

I did some research on current Python packaging practices, and per PEP 621, I decided this was the ideal place to specify my dependencies... until I realized it wasn't supported by Renovate.

https://github.com/nuztalgia/qibot/blob/7add85d229559f5caf93f6f018d984098b59ece2/pyproject.toml#L20-L32

Log excerpts from job ID #732082846 (click to expand)
DEBUG: latest repository commit
{
  "latestCommit": {
    "hash": "7add85d229559f5caf93f6f018d984098b59ece2",
[...]
DEBUG: Matched 1 file(s) for manager pip_setup: setup.py
DEBUG: Matched 1 file(s) for manager poetry: pyproject.toml
DEBUG: pre-commit is disabled
DEBUG: pyproject.toml contains no poetry section
DEBUG: Found 0 package file(s)
INFO: Dependency extraction complete
{
  "baseBranch": "main",
  "stats": {
    "managers": {},
    "total": {
      "fileCount": 0,
      "depCount": 0
    }
  }
}

setup.py

This is where I've put my dependencies for the time being, in order to maintain compatibility with Renovate - although I have to admit, this workaround feels quite unintuitive/undesirable.

https://github.com/nuztalgia/qibot/blob/59b8e6d7a70d7f6c5d5fef7c0526d0b8fafa4c7b/setup.py#L4-L16

Log excerpts from job ID #732754677 (click to expand)
DEBUG: latest repository commit
{
  "latestCommit": {
    "hash": "59b8e6d7a70d7f6c5d5fef7c0526d0b8fafa4c7b",
[...]
DEBUG: Matched 1 file(s) for manager pip_setup: setup.py
DEBUG: Matched 1 file(s) for manager poetry: pyproject.toml
DEBUG: pre-commit is disabled
DEBUG: pyproject.toml contains no poetry section
DEBUG: Found pip_setup package files
DEBUG: Found 1 package file(s)
INFO: Dependency extraction complete
{
  "baseBranch": "main",
  "stats": {
    "managers": {
      "pip_setup": {
        "fileCount": 1,
        "depCount": 7
      }
    },
    "total": {
      "fileCount": 1,
      "depCount": 7
    }
  }
}

Hopefully this helps provide some more context to support this issue. Let me know if any additional information would be useful!

@mkniewallner
Copy link
Contributor

Handling PEP 621 may actually be a bit complex to handle.

PEP 621 (and by extension, PEP 631) defines the structure for declaring dependencies, which is convenient since projects implementing the specification share a common structure.

But I believe that from a Renovate perspective, this would not really be a manager per se, but rather a common reusable structure and logic to parse, which itself is used by different managers (for instance, for libraries following PEP 621: PDM, setuptools, flit, hatch, ...).

Despite following the same structure for specifying dependencies, tools implementing PEP 621 still have their own lock mechanism (some also don't handle locking) for the lock file (both for the format and the name), since this is not covered by a PEP.

Also, since Renovate would need to run commands from the different libraries, which will most likely differ, there definitely needs to be a differentiation between the different tools.

So depending on the tool used by projects, Renovate would need a hint on the tool that was used. Something naive may be to look for a specific lock file name (since the tools will most likely have their own file name, which have a great chance of being unique).

In the future, if there's ever an accepted common format for the lock file (there was PEP 665, but it has been rejected), this may become unreliable, though, but maybe this is looking too much into what the future may become.

tekumara added a commit to seek-oss/aec that referenced this issue Jul 20, 2022
[PEP 621](https://peps.python.org/pep-0621) provides a standardised way for storing project metadata including dependencies declaratively in  pyproject.toml, instead of setup.py which is very dynamic. PEP 621 is now [supported by setuptools](https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html).

This PR uses project settings in pyproject.toml to configure setuptools to build the sdist and wheel.

The built distribution is the same, except the PKG-INFO now includes classifies, keywords, and the full text of the license.

NB: pyproject.toml dependencies are not yet supported by renovatebot (see [#10187](renovatebot/renovate#10187)) or dependabot (see [#3290](dependabot/dependabot-core#3290))
@flying-sheep
Copy link

flying-sheep commented Jul 28, 2022

I think you’re almost spot on, except for the “unreliable” part, as the future could look like this:

  1. we add support the following managers:

  2. Once a standard lock file exists we support an additional manager:

    • PEP 621’s pyproject.toml + <standard lockfile>

I don’t see how this could become unreliable or otherwise prevent action right now.

@ofek
Copy link

ofek commented Sep 3, 2022

Please keep in mind that libraries do not need lock files and will only ever use pyproject.toml. Also, this should imo be called standard.

@djmattyg007
Copy link

Please keep in mind that libraries do not need lock files

I don't think this is accurate at all. Lock files typically speed up the installation process drastically, which means they're a good idea to commit in all places. It also ensures you're using the exact same version for testing things in all environments. Libraries still have plenty of dev dependencies, and when a team is working on that one library, it's important that everyone is using the same version of those dev dependencies.

@ofek
Copy link

ofek commented Oct 1, 2022

Generally, I think it's irresponsible to use lock files for libraries rather than pulling latest to catch incompatibilities before end users do.

Lock files typically speed up the installation process drastically

Dependency resolution pales in comparison to the downloading and unpacking of artifacts, unless that is in fact necessary for dependency resolution as is the case for Python source distributions when no wheels are available.

Though this is not very applicable to libraries as there won't be potentially hundreds or thousands of dependencies like applications may have.

Libraries still have plenty of dev dependencies, and when a team is working on that one library, it's important that everyone is using the same version of those dev dependencies.

Those can just be pinned since most libraries do the approach I described and will catch issues with the latest versions of their transitive dependencies before you do 😉

@pradyunsg
Copy link

puts on the hat of "I am one of the authors of pip's dependency resolver"

@ofek is correct IMO - if libraries shipped lock files to their end users, that would be inappropriate and would be equivalent to overly aggressive pinning.

There might be a case to be made for development workflows but as @ofek noted, you typically want to get the latest versions to catch issues before end-users see them. Or, at least, end users who are consuming it via a lock file and being judicious about when/how they update things.

PEP 621’s pyproject.toml + pip-tools’ requirements.txt

I don't know much about renovate's semantics/specifics but I'll note that pip-tools already supports this workflow.

@djmattyg007
Copy link

if libraries shipped lock files to their end users, that would be inappropriate and would be equivalent to overly aggressive pinning

This simply isn’t true. Lock files are not used by dependent software. The contents of a library’s lock file has no bearing on anything outside of that project itself.

@dvzrv
Copy link

dvzrv commented Nov 30, 2022

Hi! As a user of PDM for some projects, I'm wondering whether anyone is working on implementing this for renovate (as described in #10187 (comment)).

What would be needed to make this happen?

@narwold

This comment was marked as off-topic.

@dgeschka

This comment was marked as off-topic.

@viceice
Copy link
Member

viceice commented Feb 4, 2023

Please don't post me too comments, they are too noisy. We'll delete them in future.

To get this happen somebody from the community needs to send a PR with an implementation.

it also needs a reproduction as the label states.

@sigma67

This comment was marked as off-topic.

@rarkins

This comment was marked as off-topic.

@sigma67

This comment was marked as off-topic.

@rarkins

This comment was marked as off-topic.

nijel added a commit to WeblateOrg/weblate that referenced this issue Apr 17, 2023
setuptools will stop working with this and we're not able to switch to
pyproject.toml until renovatebot/renovate#10187
djpugh added a commit to djpugh/azure_devops_artifacts_helpers that referenced this issue May 2, 2023
@secustor secustor self-assigned this May 10, 2023
@secustor secustor added status:in-progress Someone is working on implementation and removed status:requirements Full requirements are not yet known, so implementation should not be started labels May 12, 2023
@renovate-release
Copy link
Collaborator

🎉 This issue has been resolved in version 35.83.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

@secustor
Copy link
Collaborator

Thanks to Vector Informatik GmbH for sponsoring this feature!

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 16, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto:reproduction A minimal reproduction is necessary to proceed new package manager New package manager support priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others status:in-progress Someone is working on implementation type:feature Feature (new functionality)
Projects
None yet
Development

Successfully merging a pull request may close this issue.