Vendor tomli to break the circular dependency.

[jameshilliard]

Minimal vendoring implementation based on pip vendoring.

This seems to be straightforward maintenance wise and should make things a lot easier when it comes to bootstrapping flit_core.

[takluyver]

(I'm repeating myself from other issues, but I want to record this here rather than assuming people will wade through those long discussions)

I get that vendoring would make some use cases easier. But it makes other use cases more difficult, because I know some downstreams (Linux distros in particular) will unvendor it and then have to solve the same dependency cycle that the vendoring avoids.

So I'd rather look for a solution that doesn't involve vendoring. I believe I have the outline of one that only requires a minor tweak for redistributors: build flit_core (which only requires Python itself), then use tomli from source with flit_core to build tomli. You see this as some sort of eldritch abomination, and I imagine if you're used to compiled languages, using a library before building it is a weird idea. But from my perspective, this is Python imports working precisely as designed, and a perfectly reasonable approach.

If bundling packages together is acceptable to you, look at https://github.com/FFY00/python-bootstrap - whether or not that implementation is acceptable to you, look at the idea. It's essentially the same idea, just taken a step further: bundle all the fundamental packaging tools together and use them to build & install one another. That's a simple way out of any dependency cycles.

[jameshilliard]

But it makes other use cases more difficult, because I know some downstreams (Linux distros in particular) will unvendor it and then have to solve the same dependency cycle that the vendoring avoids.

If they already do that with pip it must not be that difficult for them to also do that here since they clearly have the tooling for it IMO.

You see this as some sort of eldritch abomination, and I imagine if you're used to compiled languages, using a library before building it is a weird idea. But from my perspective, this is Python imports working precisely as designed, and a perfectly reasonable approach.

It's more an issue with us not having the ability to easily handle circular toolchain dependencies(basically our entire toolchain is built from source in stages with very little base build host dependencies), if I define a circular dependency between the tomli and flit_core packages like is expected here our build system will detect that and error out as it has no way to resolve the conflict. I'm much more used to working with python than any other language, however it tends to be one of the most complex languages to cross compile due to upstream tooling being designed for only the more common use cases.

You see this as some sort of eldritch abomination, and I imagine if you're used to compiled languages, using a library before building it is a weird idea. But from my perspective, this is Python imports working precisely as designed, and a perfectly reasonable approach.

I mean using a project's bundled library to build itself is one thing, having an external circular dependency is something totally different IMO. If a package depends on itself to build we can handle that as it doesn't create a circular dependency.

bundle all the fundamental packaging tools together and use them to build & install one another. That's a simple way out of any dependency cycles.

I'm just not seeing a way to do something like this in a way that can be up-streamed. We have build/install stage isolation features that make that approach very difficult.

[jameshilliard]

This circular dependency(among with a number of other issues I've raised) is also an issue for other distros like gentoo as well:

Firstly, TOML. This is something I’ve been repeating for quite some time already, so I’ll just quickly go over it. I like TOML, I think it’s a reasonable choice for markup. However, without a TOML parser in stdlib (and there’s no progress in providing one), this means that every single build system now depends on tomli, and involves a circular dependency. A few months back, every single build system depended on toml instead but that package became unmaintained. Does that make you feel confident?

I think it's really being underestimated how big a pain point this is for distros, there's IMO absolutely no reason build tools like flit should allow circular package dependencies like this as it becomes an intractable problem for downstream integrators with packaging systems that are unable to deal with circular dependencies.

[gaborbernat]

But if those downstream integrators would add support for circular dependencies they'd solve this problem forever 🤔 for all packages. So IMHO could be worth tackling that challenge.

[jameshilliard]

But if those downstream integrators would add support for circular dependencies they'd solve this problem forever 🤔 for all packages. So IMHO could be worth tackling that challenge.

From my understanding supporting circular dependencies on our side isn't possible, buildroot uses make for dependency resolution which from my understanding is incapable of resolving circular dependencies in targets.

[takluyver]

I'm aware that the circular dependency is going to cause some difficulty for any downstream packagers. What I'd like to know better is:

• How big an obstacle is it? Major challenge or minor headache? E.g. it sounds like Spack have found a workaround without too much drama (though maybe @adamjstewart is just being modest). Fedora (@hroncok, @encukou) have also worked around it by copying tomli directly to site-packages as a special bootstrap option. Gentoo (@mgorny) are generating setup.py files, though as Michał says in his blog post, it's likely they'll need to do something else in the future.
• What possible solutions would make things easier, and what wouldn't? I've argued that Linux distros object to vendoring, and thus this isn't a good general solution, but I haven't actually heard from any Linux distros on this specific case, I'm just transferring what I've seen from other examples like pip. Would you like to special case flit_core and then build tomli (more) normally? Or special case tomli so you can build it before flit_core?

I'd like to hear this also from downstreams other than buildroot. 😉

[hroncok]

I am quite happy with Fedora's current solution in tomli, but I am not sure if it scales well in case flit_core gains more and more dependencies that use flit_core to build. In that case, we might consider treating tomli as a non-standard case instead of treating the dependencies as such.

[hroncok]

I believe I have the outline of one that only requires a minor tweak for redistributors: build flit_core (which only requires Python itself), then use tomli from source with flit_core to build tomli.

But that is not the case, is it? flit_core actually requires tomli.

[hukkin]

But that is not the case, is it? flit_core actually requires tomli.

It requires tomli only to run. But to build flit_core only Python is required.

[hroncok]

But that is not the case, is it? flit_core actually requires tomli.

It requires tomli only to run. But to build flit_core only Python is required.

Even if that is the case (and I am afraid we failed to manage to do that in Fedora, but I can check), we wold only manage to build flit_core that isn't installable before we also build tomli. And we cannot build tomli because flit_core isn't installable.

[hroncok]

We could build a special variant of flit_core that does not declare the dependency, build tomli with it (bend it so flit_core imports tomli from source), and then build flit_core reguralry. That however requires treating both flit_core and tomli as special cases.

[jameshilliard]

How big an obstacle is it? Major challenge or minor headache?

For us it seems to be impossible to resolve the circular dependency issue in a reasonably maintainable way downstream.

Gentoo (@mgorny) are generating setup.py files, though as Michał says in his blog post, it's likely they'll need to do something else in the future.

Yeah, patching in a bunch of setup.py's is probably the easiest way to handle this for us as well at the moment, but there really should be a better solution here IMO.

What possible solutions would make things easier, and what wouldn't?

For us the key thing is any solution must break the circular dependency graph.

I've argued that Linux distros object to vendoring, and thus this isn't a good general solution, but I haven't actually heard from any Linux distros on this specific case, I'm just transferring what I've seen from other examples like pip.

It's much easier for a downstream like debian to devendor a package designed to work either in a vendored or devendored configuration than for distros that can't handle circular dependencies to vendor something. I realize it's not ideal to vendor dependencies but I can't think of another way to handle this issue reliably. If there isn't a solution for devendoring pip why would there be one for other build tools? I don't see how this problem is any different from the one pip vendoring was designed to solve.

I am quite happy with Fedora's current solution in tomli, but I am not sure if it scales well in case flit_core gains more and more dependencies that use flit_core to build. In that case, we might consider treating tomli as a non-standard case instead of treating the dependencies as such.

Yeah, this will also be an issue for build and install packages(build has a fairly deep dependency tree from the looks of it at least) once they drop distutils/setuptools support as well. The situation with flit_core is just the first of these issues. We really need a proper bootstrap strategy here. For example if install drops setup.py then we would probably need install to vendor enough of its dependencies that it can build and install itself(I think this may require build and its dependency tree to be vendored, although that's not entirely clear).

It requires tomli only to run. But to build flit_core only Python is required.

For us this build vs run dependency distinction doesn't make much of a difference, if flit_core has a tomli dependency then tomli can't have a flit_core dependency due to how our dependency resolution works(since we don't really separate build vs run dependencies in a way that can handle this).

We could build a special variant of flit_core that does not declare the dependency, build tomli with it (bend it so flit_core imports tomli from source), and then build flit_core reguralry. That however requires treating both flit_core and tomli as special cases.

Yeah, for us the only way to do something like this is to maintain multiple versions of a package, but doing so would significantly complicates our toolchain bootstrap process. This would quickly become an unmaintainable mess.

[takluyver]

I am quite happy with Fedora's current solution in tomli, but I am not sure if it scales well in case flit_core gains more and more dependencies that use flit_core to build.

It's very much the plan that it won't gain more and more dependencies. The reason for making flit_core a separate package was for it to have minimal dependencies. Parsing TOML is fairly unavoidable, but other than that I don't foresee a need for any extra dependencies.

If tomli was vendored in flit_core, would you (Fedora, and any other downstreams - I know the view from buildroot):

• De-vendor it and carry on with other ways to get out of the cycle?
• Use the vendored copy to break the cycle, then de-vendor it afterwards?
• Leave it using the vendored copy in general?

As @jameshilliard mentions, it looks like we might be heading towards having a few other key packaging pieces built with Flit (installer, pep517, packaging, build). This isn't necessarily a cycle - I understand from your PackagingCon talk that Fedora has its own tooling to work with PEP 517 interfaces, so I imagine build and installer aren't so special. But for downstreams that do use these for their own packaging, some special casing will be required to prepare these initial packages. We're also trying to work out what this should look like.

[takluyver]

We could build a special variant of flit_core that does not declare the dependency, build tomli with it (bend it so flit_core imports tomli from source), and then build flit_core reguralry.

I have also wondered about publishing a flit_core_minus_tomli package, with the same code as flit_core but no dependency, just to be used to build tomli. 🤷

[jameshilliard]

This isn't necessarily a cycle - I understand from your PackagingCon talk that Fedora has its own tooling to work with PEP 517 interfaces, so I imagine build and installer aren't so special. But for downstreams that do use these for their own packaging, some special casing will be required to prepare these initial packages. We're also trying to work out what this should look like.

How would one avoid this creating a cycle here other than vendoring? I mean the current way those packages avoid creating a cycle is to have setuptools/distutils fallbacks. If install vendors everything needed to build+install build+build's dependencies then it could maybe be built first with other packages depending on it. If anything needs to be installed using install(say if build becomes a flit_core based package) then install can not depend on that package.

[hroncok]

I am quite happy with Fedora's current solution in tomli, but I am not sure if it scales well in case flit_core gains more and more dependencies that use flit_core to build.

It's very much the plan that it won't gain more and more dependencies. The reason for making flit_core a separate package was for it to have minimal dependencies. Parsing TOML is fairly unavoidable, but other than that I don't foresee a need for any extra dependencies.

That was my understanding as well and the only reason I decided to build tomli the way we do instead of trying to solve this in flit_core.

If tomli was vendored in flit_core, would you (Fedora, and any other downstreams - I know the view from buildroot):

* De-vendor it and carry on with other ways to get out of the cycle?

* Use the vendored copy to break the cycle, then de-vendor it afterwards?

* Leave it using the vendored copy in general?

Probably one of the latter 2 options. De-vendoring it afterward sounds reasonable, as long as our packaged tomli isn't much different from the vendored one (different version with different API, custom patches in vendored lib, etc.).

As @jameshilliard mentions, it looks like we might be heading towards having a few other key packaging pieces built with Flit (installer, pep517, packaging, build). This isn't necessarily a cycle - I understand from your PackagingCon talk that Fedora has its own tooling to work with PEP 517 interfaces, so I imagine build and installer aren't so special.

Well, our custom tooling still shells out to pip to build the wheel and to install it, but we would like to eventually migrate to build and install if at all possible.

[jameshilliard]

Use the vendored copy to break the cycle, then de-vendor it afterwards?

To clarify we would probably still carry tomli as a separate package for any packages that want to depend on it in cases where it wouldn't introducing a cycle, tomli would depend on a vendored flit_core for the build. We would always have to use the vendored version of tomli with flit_core I think so that it doesn't mess things up due to the one way nature of our dependency DAG.

[jameshilliard]

Well, our custom tooling still shells out to pip to build the wheel and to install it, but we would like to eventually migrate to build and install if at all possible.

Yeah, we haven't supported pip based builds/installs since it doesn't play nice with cross compilation.

[hroncok]

Also, note that Fedora's tooling uses toml to parse the pyproject.toml file and we would like to eventually switch to tomli, hence another reason we decided to treat tomli in a special way.

[adamjstewart]

I wholeheartedly agree with vendoring tomli. Setuptools also vendors its dependencies, thanks to maintainers of package managers like ourselves: pypa/setuptools#980

Spack have found a workaround without too much drama

Yes! For those who don't know, Spack is another from-source package manager designed for supercomputers. My solution in Spack was actually the reverse of what is done here. Since flit_core needs tomli at run-time, but tomli only needs flit_core at build-time, I download a copy of flit_core and use it to build tomli. Then flit_core simply has a normal dependency on tomli. This is only possible because none of these packages are compiled, and I can simply add the source code to my PYTHONPATH to get things working. We do the same thing when bootstrapping pip/wheel/setuptools, as inspired by Nix. See here for more discussion. I actually wish pip would vendor its own deps so I could avoid this.

I get that vendoring would make some use cases easier. But it makes other use cases more difficult, because I know some downstreams (Linux distros in particular) will unvendor it and then have to solve the same dependency cycle that the vendoring avoids.

I think there are different levels of guiltiness when it comes to vendoring. Some packages like PyTorch/Tensorflow vendor their dependencies even when they don't need to, and rely on very specific commits (not stable releases) of their dependencies. This is a nightmare for package managers. In the case where vendoring happens to allow a package to build from source without cyclic deps, this is okay in my mind. Spack does the same thing, vendoring its own dependencies so that users don't need to install anything to get a working package manager.

I think the important thing is that:

• vendored deps don't pollute the namespace (so installing flit_core won't cause tomli to show up in site_packages, only in a subdir of flit_core)
• stable releases are used and updated from time-to-time to prevent bugs/security vulnerabilities
• vendored copies are included in tarballs on GitHub and PyPI and don't rely on cloning git submodules to get

[takluyver]

Thanks all. It sounds like there's maybe less objection to vendoring than I had thought, though I'd still like to hear from other downstreams - if you know of any people who might be working on this in other environments, do ping them.

It's only tomli that I'd consider vendoring in flit_core - not build, installer or their dependencies. If you want to go all in on the bundling approach, look at https://github.com/FFY00/python-bootstrap (for inspiration, not necessarily to use as is).

What else can we do to make it feasible to bootstrap these packaging tools? Ideas I've had so far:

• flit_core (and maybe tomli) could ship an install_bootstrap.py script in the sdist (e.g. Add bootstrap install script for flit_core #481), so you can install it before having installer available.
• flit_core could have a minimal CLI so you could use python -m flit_core.wheel as a replacement for python -m build until you have build (if build & its dependencies do all end up using Flit).

[jameshilliard]

It's only tomli that I'd consider vendoring in flit_core - not build, installer or their dependencies.

Agree, if anything it would make sense to vendor flit_core and its tomli dep in build or something using build would maybe vendor both.

[jameshilliard]

flit_core could have a minimal CLI so you could use python -m flit_core.wheel as a replacement for python -m build until you have build (if build & its dependencies do all end up using Flit).

Maybe it would make sense to have a vendored version of flit_core in build? I think rather than spreading out build functionality in a lot of projects which gets complex just have build system projects that use flit_core vendor flit_core(maybe with runtime perference for using the non-vendored version using import fallbacks) and any other deps they need to be able to do basic build and installs. This way for downstream integrators it would be clear which package you download first is(the one that vendors everything needed to get started with), like how you can download setuptools as one package which is capable of building and installing.

[gaborbernat]

build knows nothing about flit_core why would it want to vendor it? 🤔

[pradyunsg]

I think rather than spreading out build functionality in a lot of projects which gets complex just have build system projects

That is fundamentally incompatible with the model that Python's packaging story has adopted with https://www.python.org/dev/peps/pep-0517/ -- packages should be able to use any sort of custom build-backend from any Python package they want, as long as it is a build-time dependency declared in the pyproject.toml file. There's no specific "blessed" build backends (other than setuptools, because legacy reasons) and it should be possible to use any other Python package as a build backend, assuming it implements the required API.

[adamjstewart]

Similarly to how there isn't a single "blessed" build backend, I don't think there is a single "blessed" build frontend either. I don't think build/installer are very popular compared to pip. I'm currently overhauling the way Spack builds/installs Python packages, so I was looking at using build/installer to do this, but both of them have a ton of dependencies making bootstrapping an absolute nightmare. I ended up using pip instead. This isn't directly relevant to the conversation at hand, so I don't want to sidetrack this PR, but if anyone has any strong thoughts about build/installer vs. pip, please comment on spack/spack#27798.

[encukou]

FWIW, vendoring and installing from wheels have the same issue: there's a second "source" that needs to be independently audited and maintained. (Assuming wheels aren't used as the "main" source, which is often problematic as they omit tests or documentation.)

---

flit_core (and maybe tomli) could ship an install_bootstrap.py script in the sdist (e.g. #481), so you can install it before having installer available.

This looks similar to the Fedora approach of unpacking directly to site-packages, except it's not Fedora-specific. Nice! And it also skips dependency checking, so it removes the need for a flit_core_minus_tomli.

[mgorny]

I've already switched Gentoo to copy the files from the wheel directly. Right now the thing that sucks the most is that we have to download two separate archives with some common files: wheel to get the dist-info, and github archive to get tests.

[takluyver]

Maybe it would make sense to have a vendored version of flit_core in build?

I'm not involved in build, so maybe the authors of that would be more amenable to vendoring all of its dependencies, but I would be surprised if they do, and I don't think it's necessary - I think we can provide a reasonable bootstrapping option to build packages with flit_core before build itself is set up.

I feel like I'm repeating myself, but if you want a bundle of all of these projects together to simplify bootstrapping, you can make that yourself. It doesn't have to be an official release of one of these projects. The python-bootstrap repo shows how you can use the projects together to get them all installed.

I don't think build/installer are very popular compared to pip. I'm currently overhauling the way Spack builds/installs Python packages, so I was looking at using build/installer to do this, but both of them have a ton of dependencies making bootstrapping an absolute nightmare.

(Responded on the Spack issue - there is a reason they exist, but if it's easier for you to use pip, that's fine)

This [bootstrap install script] looks similar to the Fedora approach of unpacking directly to site-packages, except it's not Fedora-specific. Nice! And it also skips dependency checking, so it removes the need for a flit_core_minus_tomli.

Thanks - it sounds like that would be useful for at least some use cases. To be clear, this would work by building a wheel and unpacking it again. Let's take any further discussion of that idea to #481. 🙂

I've already switched Gentoo to copy the files from the wheel directly. Right now the thing that sucks the most is that we have to download two separate archives with some common files: wheel to get the dist-info, and github archive to get tests.

Are you talking about flit_core specifically, or a broader set of packages? For flit_core, it should be possible to build the wheel from the Github archive (or the sdist) with no dependencies besides Python (see build_dists.py), so there should be no need to download a wheel as well.

[jameshilliard]

build knows nothing about flit_core why would it want to vendor it? 🤔

I guess it would probably make more sense to vendor flit_core and build in installer, at least by the time installer drops distutils/setuptools compatibility. This way distributions can use installer in a way other users use pip for a single stage bootstrap.

There's no specific "blessed" build backends (other than setuptools, because legacy reasons) and it should be possible to use any other Python package as a build backend, assuming it implements the required API.

Additional backends are only a problem if they have circular dependencies, I think the important part is that the build frontends like installer are fully functional without any circular dependencies along the lines of pip so that they can then be used to directly bootstrap(build+install) additional build dependencies that may be needed.

I don't think build/installer are very popular compared to pip. I'm currently overhauling the way Spack builds/installs Python packages, so I was looking at using build/installer to do this, but both of them have a ton of dependencies making bootstrapping an absolute nightmare.

I think if installer vendors all its dependencies using it would be a lot more viable, right now it already mostly works without vendoring but that's because for the most part its dependencies can be installed with distutils/setuptools(although I had to hold back tomli to a version before distutils fallbacks were removed). But if installer moves to requiring flit_core then it really should also start vendoring the flit_core dependency tree.

FWIW, vendoring and installing from wheels have the same issue: there's a second "source" that needs to be independently audited and maintained. (Assuming wheels aren't used as the "main" source, which is often problematic as they omit tests or documentation.)

Vendoring should be done without namespace pollution, meaning the vendored dependency should only be directly used by the package that vendored it. This should be present in the main repo/sdist along the lines of pip so that wheels from pypi aren't needed for bootstrapping.

flit_core (and maybe tomli) could ship an install_bootstrap.py script in the sdist (e.g. #481), so you can install it before having installer available.

I think you get better deduplication of build+install functionality by vendoring everything needed to build and install installer inside installer, and this much more closely matches how things currently work with distutils/setuptools, if installer has no external required unvendored dependencies then it can just build and install itself.

Right now the thing that sucks the most is that we have to download two separate archives with some common files: wheel to get the dist-info, and github archive to get tests.

Yeah, we really don't want to have to do something like this.

I feel like I'm repeating myself, but if you want a bundle of all of these projects together to simplify bootstrapping, you can make that yourself. It doesn't have to be an official release of one of these projects. The python-bootstrap repo shows how you can use the projects together to get them all installed.

So the problem with that approach and why upstream vendoring is a lot better is that you would get namespace pollution if you just combine the installs naively like in python-bootstrap, meaning the special cased bundled tomli dependency would conflict with say the unbundled tomli that you would use as a normal package dependency, using the namespace isolating style of vendoring like in this PR tomli gets properly isolated for use by flit_core and can't conflict with other tomli installations.

[mgorny]

I've already switched Gentoo to copy the files from the wheel directly. Right now the thing that sucks the most is that we have to download two separate archives with some common files: wheel to get the dist-info, and github archive to get tests.

Are you talking about flit_core specifically, or a broader set of packages? For flit_core, it should be possible to build the wheel from the Github archive (or the sdist) with no dependencies besides Python (see build_dists.py), so there should be no need to download a wheel as well.

Sorry, I didn't look to which bugtracker I am replying ;-). I was talking of tomli for now, as setuptools started depending on it. Flit I'm still building via pp2sp and setuptools but I'm getting really tired of having to repeat all the awful hacks (like implicit src/ guessing), so I guess I'll start trying to use "build thyself" logic when we finally get a single working PEP517 installer.

[jameshilliard]

Sorry, I didn't look to which bugtracker I am replying ;-). I was talking of tomli for now, as setuptools started depending on it.

Isn't setuptools adding a properly vendored version of tomli in #2924?

[jameshilliard]

By the way I just noticed that PEP-517 explicitly prohibits any sort of dependency cycles in the build requirements section:

This PEP places a number of additional requirements on the "build requirements" section of pyproject.toml. These are intended to ensure that projects do not create impossible to satisfy conditions with their build requirements.

• Project build requirements will define a directed graph of requirements (project A needs B to build, B needs C and D, etc.) This graph MUST NOT contain cycles. If (due to lack of co-ordination between projects, for example) a cycle is present, front ends MAY refuse to build the project.
• Where build requirements are available as wheels, front ends SHOULD use these where practical, to avoid deeply nested builds. However front ends MAY have modes where they do not consider wheels when locating build requirements, and so projects MUST NOT assume that publishing wheels is sufficient to break a requirement cycle.
• Front ends SHOULD check explicitly for requirement cycles, and terminate the build with an informative message if one is found.

Note in particular that the requirement for no requirement cycles means that backends wishing to self-host (i.e., building a wheel for a backend uses that backend for the build) need to make special provision to avoid causing cycles. Typically this will involve specifying themselves as an in-tree backend, and avoiding external build dependencies (usually by vendoring them).

Based on my reading of this PEP-517 provision any packages with flit_core as a build requirement would not be PEP-517 compliant unless flit_core vendors tomli.

[jameshilliard]

I wrote a basic build requirement cycle detector for build in #415. It should hopefully prevent build tool package developers from accidentally violating the PEP-517 no circular dependencies requirement.

[takluyver]

OK, accepted the principle of vendoring. To implementation.

My thoughts are basically: let's keep it as simple as possible.

[takluyver]

Let's revert this and ensure tomli is installed for tests in Flit (not flit_core).

[takluyver]

I'm not keen on involving an extra automation tool (nox) and extra bits of CI logic just for this.

TBH, I think Pradyun's vendoring tool is overkill here. I can absolutely see why you'd want it for pip, where there are 25 vendored packages already, but we're only vendoring one, there should be no need to vendor more, and I'd like to make it clear that it's just one. Hopefully even that one will be unnecessary in a few years (if a TOML parser is added to the standard library).

So I think it's easier to update it by hand from time to time than to manage extra code for updating it.

[hukkin]

Version 1.2.3 has been released now. It fixes one bug. We may want to update Tomli to that.

[takluyver]

I've opened #492 to do this but without the extra bits of tooling and automation - it's just one package, so I want to keep it simple.