Standard Python support

[flying-sheep]

There’s exactly two standard ways of specifying dependencies, none of which dependabot supports.

1. PEP 517’s prepare_metadata_for_build_wheel().

   The legacy way to specify Python dependencies is requirements.txt. The standard way is having a PEP 517 build system configured. That build system can optionally define a function called prepare_metadata_for_build_wheel. If that function exists, the dependencies can be obtained (from the metadata) without building a wheel, otherwise build_wheel() needs to be called.

   Code to just do the above would look like:

   import build
   deps = build.ProjectBuilder("path/to/project").get_dependencies("sdist")
   # or
   deps = build.ProjectBuilder("path/to/project").get_dependencies("wheel") - {"wheel"}

2. PEP 631 a new standardized way to specify dependencies in a Python project (now merged into PEP 621).

   It’s supported by flit and a bunch of other new build backends. So if you want to continue your (incomplete) way of just parsing files and not calling into Python, this needs to be supported.

[flying-sheep]

This should be pretty straightforward (if one knows ruby, which I don’t). You already have

1. a format preserving TOML parser (for poetry support)
2. a parser and serializer for the PEP 508 dependency specifier language (for requirements.txt support)

So you just have to read the TOML lists project.dependencies and project.optional-dependencies.* instead of the toml tables tool.poetry.dependencies and tool.poetry.extras.*, and then use the PEP 508 handling logic instead of the poetry version specifier logic

[dreinon]

I just got a dependabot PR of my project using pdm! Hope this has been finally accepted 🙂
EDIT: False alarm, it's a dependency from github actions 🙁 , too good to be true haha

[drewcassidy]

now that setuptools supports PEP621, It would be nice to see this

[drewcassidy]

So you just have to read the TOML lists project.dependencies and project.optional-dependencies.* instead of the toml tables tool.poetry.dependencies and tool.poetry.extras.*, and then use the PEP 508 handling logic instead of the poetry version specifier logic

I would say to read build-system.requirements too. some of those things may be pinned too (like a project using pybind11).

[ofek]

I would say to read build-system.requirements too

Minor correction: build-system.requires

[jorgepiloto]

I am surprised after finding that dependabot was able to parse a pyproject.toml using Poetry as the build-system, see https://github.com/pyansys-testing/dependabot-pyproject/pull/3/files

[jorgepiloto]

Both required and optional dependencies were parsed

[jorgepiloto]

However, when using Flit as the build-system, dependabot did not opened any PR.

[vnmabus]

It seems that this issue also causes projects moving towards pyproject.toml adoption to lose their list of dependents, unless the name of the project is also repeated in setup.py.

See https://github.saobby.my.eu.orgmunity/t/bug-project-not-showing-network-dependents/3161 for examples.

[jenstroeger]

@jorgepiloto @vnmabus exactly my problem after moving to pyproject.toml and Flit.

To be honest, I am surprised that @dependabot doesn’t support such a fundamental and standardized process, and quite concerned that this issue isn’t being addressed. We’re actually now exploring alternatives, e.g. @renovatebot (link).

[jurre]

Hey, thanks for the pings on this! It's definitely on our radar and we want to support this, but I can't share a timeline as to when we'll be able to ship support for this across both Dependabot and the Dependency Graph.

[davidism]

I've left some comments on the PR that's about to get merged. #5661 I'm worried that only supporting pins in pyproject.toml will encourage users to put pins there, which will be fine for applications but really bad if libraries pin like that. Would really like to see support for lock files, even though they're not standard yet, since there's already non-standard Poetry support.

[deivid-rodriguez]

We've shipped preliminary support for updating dependencies following PEP621 in pyproject.toml files with #5661.

Of course there may be bugs and missing features, but I'm closing this main issue now and will track any future enhancements as separate issues.

Thanks!

[zhan9san]

I've left some comments on the PR that's about to get merged. #5661 I'm worried that only supporting pins in pyproject.toml will encourage users to put pins there, which will be fine for applications but really bad if libraries pin like that. Would really like to see support for lock files, even though they're not standard yet, since there's already non-standard Poetry support.

@davidism
Thanks for your time
Please allow me to ask a stupid question.

Why it is bad if libraries pin like that?
Why it's better to support for lock files?

[ofek]

@zhan9san https://iscinumpy.dev/post/bound-version-constraints/

But there's no cause for concern:

[...] we not only support full pins, but also other perfectly valid ways of restricting dependencies. Just in case there's some misunderstanding there.

[tiangolo]

I just came back here to say THANK YOU @deivid-rodriguez and everyone involved! 🤗

I'm now getting Dependabot upgrades in my projects with pyproject.toml including FastAPI and others 🎉

[deivid-rodriguez]

Thank you for the kind words @tiangolo, we're glad this is turning out useful for your projects 🎉.

[annarosenthal]

👋 Hi from the dependency graph team. We don't have a public repo, but we do have this work prioritised and I can update here once it's fixed.

Despite the issue being closed, still wanted to share that the changed needed from the Dependency Graph side has been shipped.

[skshetry]

@annarosenthal, that’s great to hear. But it still says “pyproject.toml has no dependencies or is too large to display” for us. See iterative/dvc for example.

Also, does this mean that we can now use dependency-review-action with pyproject.toml files?

[ofek]

@annarosenthal Awesome! Are there docs?

[annarosenthal]

@annarosenthal, that’s great to hear. But it still says “pyproject.toml has no dependencies or is too large to display” for us. See iterative/dvc for example.

Also, does this mean that we can now use dependency-review-action with pyproject.toml files?

Thanks for flagging. We looked into it and this is caused by a discrepancy in assumptions we made for different/older PEP standards. Apologies about it. However, I filed a bug issue internally asking to update this issue once the fix ships.

[annarosenthal]

@annarosenthal Awesome! Are there docs?

What kind of docs are looking for, in particular? We have this general doc - please let me know what kind of specific questions and/or topics aren't covered there that you still have questions about.

[akkornel]

@annarosenthal Awesome! Are there docs?

What kind of docs are looking for, in particular? We have this general doc - please let me know what kind of specific questions and/or topics aren't covered there that you still have questions about.

Hello! From my perspective, I do see one source of confusion, in the Supported package ecosystems section of the About the dependency graph document.

The Supported package ecosystems section has a list of supported package managers. For Python, there are two: pip and Python Poetry. Today, the pyproject.toml file is only listed as a supported format for Python Poetry, and that is the source of my confusion: Should pyproject.toml now also be listed as a supported format for pip? And should it be a recommended format for either pip and/or Python Poetry?

[flying-sheep]

I think the idea behind the docs is simply: “we recommend using a lockfile”. I’m not sure why that is – It’s not a good idea to pin versions for Python libraries and for libraries, I rather run tests with the newest version of everything instead of some older lockfile version, because that’s what people get when they install my library.

requirements.txt is a convention, not a standard. Some people use it for abstract dependencies (reading it in setup.py and writing its contents to the package’s metadata), others use it as a lockfile, e.g. via

pip-compile --generate-hashes pyproject.toml --output-file=requirements.txt

But as long as dependabot recommends lockfiles, and Python doesn’t have a standard lockfile format, it makes sense to say “we recommend using poetry.lock, requirements.txt AS LOCKFILES”, while of course the recommended format for Python package metadata remains PEP 621.

---

Maybe the whole segment should be reworded to be more clear.

Our recommendations:

• in addition to abstract dependencies, also use a lockfile
• avoid code-as-configuration

We support the following formats:

Language	Abstract dependencies	Lockfile	Code-as-configuration
Python	pyproject.toml, pipfile	requirements.txt[‡], poetry.lock, pipfile.lock	setup.py
Node	package.json	package-lock.json, yarn.lock
…

[‡] requirements.txt can be used as a lockfile via pip-tools. It is sometimes also used for abstract requirements, but using pyproject.toml is preferable.

[ofek]

We have this general doc

That helps, thanks!

[courtneycl]

👋 Hi folks -- with the latest changelog you should be seeing the expected dependencies from your pyproject.toml files in your dependency graph insights. Do let me know if you experience any issues.

Thank you for your feedback on the docs! I'll share it with the docs team.

[cpswan]

I may be in a small minority doing this, but we only use pyproject.toml and poetry.lock to generate a requirements.txt that has pinned SHAs (to satisfy dependency pinning for OpenSSF Scorecards).

When a dependency gets bumped I'm seeing Dependabot PRs that update the poetry.lock but to get the requirements.txt (which is actually used for pulling in dependencies) into sync I then have to manually run:

poetry export --format requirements.txt --output requirements.txt

Yes, I guess I could change my workflow to use poetry for the install (though I think there's potentially a bootstrapping problem there). But it would also be nice if Dependabot could just keep poetry.lock and requirements.txt in sync for me.

Examples:

A Dependabot PR that modifies poetry.lock but leaves requirements.txt untouched
The PR for the manually generated changes to requirements.txt (feels like this could easily be done by Dependabot too)
The workflow that runs python3 -m pip install --require-hashes -r tools/requirements.txt

[patrick91]

👋 Hi folks -- with the latest changelog you should be seeing the expected dependencies from your pyproject.toml files in your dependency graph insights. Do let me know if you experience any issues.

Thank you for your feedback on the docs! I'll share it with the docs team.

Does this mean we can remove the stub setup.py from project using poetry?

Will this also for PDM? 😊