Thoughts about alternate solutions to extraneous outputs

[henryiii]

I would like to do a proper study of what happens in several situations. Let's say I have a package that depends on another package to build, called "wheelless", maybe picking up headers or something, and that other package does not have a wheel. If I type pip wheel ., pip will build all wheels required, including the wheel I don't have control over, and put it in the "wheelhouse". That's what it needs for installing, after all, and that's what it's for, not really for package authors building a wheel. To avoid this, we add --no-deps, even though that breaks all usages of setup_requires. So, the question is, what happens in the following situations:

1. install_requires="wheelless", but no setup_requires. pip install . is broken, since it builds before installing. You have to manually install wheelless before building or installing.

2. No install_requires, setup_requires="wheelless". This downloads the package but doesn't install it anywhere, making it completely useless.

3. install_requires="wheelless", setup_requires="wheelless". Now pip install . works. pip wheel . works too, of course, since it's part of an install, but it puts the "wheelless" wheel into the wheelhouse. pip wheel . --no-deps does not work, since it disables this mechanism, so cibuildwheel fails, and this is the origin (and only place it is useful) of the suggestion CIBW_BEFORE_BUILD: pip install ., which builds the package twice, doubling your CI resources. (unless the wheels are reused from the old cache?). Note that this is a very tacky workaround on setuptools's part, it has to run setup.py twice, it can't add anything and also use it in setup.py itself, it can't control setuptools' version, etc., so it was deprecated and replaced with pyproject.toml.

4. pyproject.toml contains wheelless. This is actually the only solution that does not install wheelless in the target environment, but only has it present in the isolated build environment. I'm not completely sure this does not also end up with wheelless in the output folder, but in my projects I seem to just upload the wheel dir with no problems, so I am almost sure it does not. This "just works" for cibuildwheel, since it's not disabled (--no-build-isolation/PIP_NO_BUILD_ISOLATION).

---

It would be nice to improve the experience for 3); one option would be to use pypa-build, which should not put build requirements into the output dir, I think.

Another option would be to manually filter, but I think that's too hard - a package can specify more than one output, which would be valid, names are not guarantied, etc.

What about adding --only-binary=:all: instead, perhaps? Instead of not working, this will just not work if it has to build an unexpected wheel, and users will know what package they should pre-install when it fails. For the most common dependencies (universal wheels), there's no reason to completely disable them (pip is smart enough to only put new wheels in the wheelhouse, not ones it downloaded). I haven't tried this, though, it would be quite silly if it expected the current project to have a wheel!

[YannickJadoul]

I'm not getting the issue with 1., actually. pip wheel --no-deps . is not supposed to install anything, so install_requires is rightfully ignored. When the user wants to run tests with CIBW_TEST_COMMAND, pip install some_kind_of.whl is run, which does include downloading, building, installing dependencies. So this is mainly an issue with the package config, not with cibuildwheel?

[henryiii]

There is no issue with 1, it's just flat out incorrect and broken. I was just listing out the possibilities, 3 is the only one that is potentially surprising in cibuildwheel now, due to our --no-deps injection. CIBW_BEFORE_BUILD: pip install . would break in this case. Users would always have to list the requirements in CIBW_BEFORE_BUILD.

[YannickJadoul]

Right, OK. And what's wrong with 2? In principle, you might need a package to build, but not to run? (thinking of Cython; I know that's probably more complicated)

[henryiii]

The problem with 2 is it doesn't work. If you download a package before (re)running setup.py, what good does that do if it's not installed? You can't use it, it's still zipped up and sitting around somewhere. pyproject.toml does the correctly, as it creates a venv, installs build requirements there, builds there, then throws it away - build requirements are not kept around.

[YannickJadoul]

But that's an issue with setup_requires, again? Does that differ from 3.? install_requires only installs such a package on pip install, or not?

[henryiii]

In 2., pip install . is already broken if you depend on something in install_requires during the setup and build phase. cibuildwheel does not interfere or change anything, and CIBW_BEFORE_BUILD: pip install . will not work, it will just move the breakage earlier in the cibw process. If you don't depend on it during the build, then it doesn't matter if you install it just for a build later.