Dependency graph support for statically-defined Python dependencies

[daneah]

Despite continued wide usage of setup.py to define dependencies for Python packages, the setup.cfg file is currently the preferred and officially recommended approach to defining package dependencies statically when possible.

Looking forward, PEP 621 has outlined the use of pyproject.toml to statically define metadata in a tool-agnostic way. From the PEP:

To speak specifically to the motivation for static metadata, that has been an overall goal of the packaging ecosystem for some time. As such, making it easy to specify metadata statically is important. This also means that raising the cost of specifying data as dynamic is acceptable as users should skew towards wanting to provide static metadata.

Requiring the distinction between static and dynamic metadata also helps with disambiguation for when metadata isn’t specified. When any metadata may be dynamic, it means you never know if the absence of metadata is on purpose or because it is to be provided later. By requiring that dynamic metadata be specified, it disambiguates the intent when metadata goes unspecified.

The static and standardized format setup.cfg and pyproject.toml provide should prove amenable to parsing. Supporting these files as first-class citizens of the dependency graph will both capture projects that are currently missed as well as remove a mild friction that would push package authors toward using setup.py because it happens to be supported. This will also give package authors using statically-defined dependencies increased visibility into the reach of their work.

Package authors can't work around this limitation by e.g. reading dependencies from requirements.txt, because setuptools doesn't support the file: syntax for the install_requires field.

[jefftriplett]

Some large popular Python projects, like Flask, are coding around GitHub's limits instead of supporting the better Python standard for no benefit other than GitHub's. See here for an example: https://github.com/pallets/flask/blob/main/setup.py#L3-L16

[daneah]

It looks like pyproject.toml is now supported specifically for Poetry, but setup.cfg and full PEP 621 support will still be important for package maintainers.

[mauvilsa]

Many people comment in dependabot/dependabot-core#2133 thinking that is what is needed to add setup.cfg support to the dependency graph, though it isn't. It seems it is a highly wanted feature.

[sethmlarson]

Dropping in to report that Flit is also not supported. urllib3 had it's "Used By" and "Sponsor who you depend on" support removed due to this. Our team had to work around this by writing fake metadata into our setup.py which immediately restored the features for the project.

This seems like a fairly widespread issue for the Python ecosystem, for example the package wheel is being reported as an archived fork of the actual project (ref) due to not supporting setuptools-scm:

[liZe]

Dropping in to report that Flit is also not supported.

Dependencies can now be listed in a standard way in pyproject.toml, as described by PEP 621. They don’t require different entries for flit, poetry or setuptools anymore.

[daneah]

This is great news; when I wrote this up, I know that at least setuptools had not yet built support but it looks like it does as of around ~2022-03-24. This would be the(?) thing to support in the dependency graph going forward, then.

[real-yfprojects]

Support for pyproject.toml in setuptools is still experimental:

Support for declaring project metadata or configuring setuptools via pyproject.toml files is still experimental and might change in future releases.

[jenshnielsen]

Note that the settings as defined by pep-621 i.e. the [project] section is no longer considered experimental in setuptools as of setuptools 62.5

[annarosenthal]

Greetings from Dependency Graph team! Trying to get ramped up on the issue and poetry/PEP developments quickly while simultaneously hoping to roll out the most accurate fix for all.

Would anyone be kindly able to confirm that the most optimal way to extract a package name would be by keying on either project.name (following this doc) or tool.poetry.name (following this doc)? Or should either one take precedence?

Thank you.

[henryiii]

For context: the items in build-system.requires are build-time dependencies. They are only required to build the package, not to install a pre-built package. As someone who works on build-systems, I'd love it if the dependency graph supported this too; that would allow us to see how many people are using poetry, hatchling, modern setuptools, etc! :)

For setup.cfg, yes, it's an older way to specify (install) dependencies, but TOML is easier to parse from other languages (.cfg is an INI format pretty specific to Python's confparser), and over time, I'd expect the reason for setup.cfg to exist will go away. There are a lot of packages using it, so it would be nice to support, but much less important than pyproject.toml in the long run. (Admittedly, supporting it is a lot easier than supporting setup.py probably was!)

[mauvilsa]

@annarosenthal your question only mentions poetry and PEP. Will support for setup.cfg also be added?

@annarosenthal will setup.cfg be supported?

[jenshnielsen]

Note that while setup.cfg is not yet deprecated it appears that it is still the plan to do that eventually. See pypa/setuptools#3214

[mauvilsa]

Even though setup.cfg might eventually be deprecated and removed from setuptools, there is no known date for this. And when and if this happens, there will still be a significant amount of repositories which use setup.cfg for which there are not enough maintainers to bother switching to pyproject.toml. Thus, it is fair to ask if setup.cfg will be supported.

[eli-schwartz]

I would imagine that setuptools would simply "eternally deprecate" the use of setup.cfg, if so, and never actually remove it. They would not want to break all those projects either, and for setuptools, breaking those projects doesn't mean "github won't display a dependency graph" -- it means "old but working projects whose authors have lost interest, left tech, or died, cannot be used anymore at all, because they won't build".

[NickleDave]

Hi @annarosenthal you all added this, right?
https://github.blog/changelog/2023-02-13-dependency-graph-supports-the-python-pep-621-standard/

Just noticed I suddenly have "Used by" on repos.
Thank you!!!

[annarosenthal]

Hi @annarosenthal you all added this, right? https://github.blog/changelog/2023-02-13-dependency-graph-supports-the-python-pep-621-standard/

Just noticed I suddenly have "Used by" on repos. Thank you!!!

My team did, indeed. Glad you find it useful! 🚀 Happy + safe coding.

[daneah]

@spaltrowitz can you consider removing this as the answer? I think there is still appetite for support for setup.cfg as originally called out in my opening of the discussion, though I very much appreciate support for pyproject.toml! Sorry, I see I have the power; the button was easy to miss.

[jeffwidman]

@daneah it looks like setup.cfg is essentially deprecated?

• Eventually deprecate setup.cfg with automatic conversion to pyproject.toml pypa/setuptools#3214
• I see many of the more popular python libs are migrating away from setup.cfg and standardizing on pyproject.toml

Given that, should this be the accepted answer again?

[daneah]

@jeffwidman I think pyproject.toml has absolutely won this, but there are still a massive number of projects (some major) that are using setup.cfg out there, so there's an unconnected part of the dependency graph which impacts all GitHub users interested in this information. I'm very much aspiring to pyproject.toml for everything within my control, but still would love this request to get a solution.

[eli-schwartz]

Anecdotally I'm not especially interested in migrating an existing project (meson) off of setup.cfg as it introduces churn without adding new features, and raises the minimum required version of setuptools.

If I was starting a new project, sure, I'd use it.

[sgbaird]

Incidentally, one of the major Python template generators, PyScaffold, uses setup.cfg which means none of the dependency graphs for the repos that use it get recognized.

pyscaffold/pyscaffold#718 (comment)