Deterministic builds

[joerick]

An attempt at getting deterministic builds in cibuildwheel.

• Pin versions of pip, setuptools, virtualenv etc. installed on macOS and windows
• Create separate constraints for python 2
• Pin manylinux images (depends on Image versioning: Tag each docker build with a unique id, and push that to quay.io manylinux#440 ✅)
• Pin/vendor/stop using get-pip.py
• Add tests

...anything else for that list?

Fix #239

[Czaki]

add https://dependabot.com/ to repository?
dockerhub allow to create free docker repository. It can be used as mirror?

[joerick]

add https://dependabot.com/ to repository?

I see dependabot around on other repos and I'm not a big fan of the sea of PRs it generates - makes it look like a lot of noise. I was wondering if version bumps could be integrated into the release process - i.e. run bin/update_constraints, push to github, check CI passes, then release.

But that does have its pitfalls - if that CI fails, then I've got to revert, or do some debugging on master, and the PRs that were merged weren't tested against the latest constraints...

So maybe it's the best approach - probably I just need to get used to seeing those PRs and ignoring them!

[joerick]

dockerhub allow to create free docker repository. It can be used as mirror?

My PR got merged - Manylinux is pushing tags for each version now, so we don't need to bother with that :)

[Czaki]

Ok. Few remarks.

1. Pip is working in transactions, you can use new version of package after all installations are finished. So I think, that install of pip and setuptools should be perform in different command that installation of rest of packages. So thin it is better to split dependecies.in on two files.
2. To easier use of some dependecy boot i think that it is better to pin version by == in dependecies.in. Then It will create PR on bump version which allow to fast find which upgrade breaks system build
3. auditwheel depends on wheel api which is not stable. So this package is pinned to given version of wheel and upgrading wheel may break build system. If we do not want to wait with releases on upgrade of manylinux images, then there is need to allow inconsistency between builds package version on different systems...
4. I do not think that installing auditwheel and delocate on all system is good idea... Eg. none test if delocate work properly on non macos system, and if it do not create any problem.
5. I think that it is good idea to pin get_pip.py script to some version (working one), then install priper version pinned by pip-compile. There are cases when get_pip.py returns non working pip version.

[joerick]

Thanks for taking a look!

1. Pip is working in transactions, you can use new version of package after all installations are finished. So I think, that install of pip and setuptools should be perform in different command that installation of rest of packages. So thin it is better to split dependecies.in on two files.

I'm not sure why splitting this is important (If the original pip is broken, it won't be able to update itself). The goal here is to hand off to user code with our specified versions installed.

2. To easier use of some dependecy boot i think that it is better to pin version by == in dependecies.in. Then It will create PR on bump version which allow to fast find which upgrade breaks system build

Assuming you mean dependabot... yes this could well be the best way to do it!

3. auditwheel depends on wheel api which is not stable. So this package is pinned to given version of wheel and upgrading wheel may break build system. If we do not want to wait with releases on upgrade of manylinux images, then there is need to allow inconsistency between builds package version on different systems...

Thanks, I didn't know that about auditwheel. In fact, auditwheel already locks wheel down to a specific version in their setup.py (in this case, their requirements.txt due to the use of pbr).

In that case, we can let pip-compile figure out the correct version of wheel to pin - it should follow the dependency tree and do the right thing. So we won't pin that in constraints.in.

4. I do not think that installing auditwheel and delocate on all system is good idea... Eg. none test if delocate work properly on non macos system, and if it do not create any problem.

These are constraints files, not requirements files, so they won't be installed unless specifically mentioned.

5. I think that it is good idea to pin get_pip.py script to some version (working one), then install priper version pinned by pip-compile. There are cases when get_pip.py returns non working pip version.

Agreed.

[Czaki]

These are constraints files, not requirements files, so they won't be installed unless specifically mentioned.

Ok, I need to learn a lite about this.

Pip is working in transactions, you can use new version of package after all installations are finished. So I think, that install of pip and setuptools should be perform in different command that installation of rest of packages. So thin it is better to split dependecies.in on two files.
I'm not sure why splitting this is important (If the original pip is broken, it won't be able to update itself). The goal here is to hand off to user code with our specified versions installed.

I wrote this because I misunderstund how constrains work.

[joerick]

Pinned dependencies are different on python2 versus python3! some packages e.g. setuptools have dropped python2 support. I'll have to generate a different constraints.txt for python 2.

[joerick]

Python 2 is being awkward, and there doesn't seem much point in continuing to write around it since #265 is nearly here, so I'll pause this until it does.

[mattip]

Something else must be involved, since the commands you pasted worked for me. To extract and run pypy, you can download it from https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-win32.zip.

As for get-pip.py vs. ensurepip: eunsurepip is supported from 2.7.9+ and 3.3+. I would advise using it over get-pip since it allows the python implementation to ship a known-good version of pip and setuptools.

[Czaki]

I'm mainly linux user and I do not know how to deal with this error:

Have you any suggestion?

[mattip]

Do you have any PYTHON related environment variables set like PYTHONPATH or PYTHONHOME?

[Czaki]

I found that I have Python 3.7 in PATH. But I do not see it in enviroment variable editor. Did you know where it can be also hidden?

[mattip]

from your DOS command window you can do

set PY

which will print any set environment variables starting with PY

[mattip]

Please try to copy-paste instead of screen shots

[mattip]

You can reset the PATH for just this DOS session with

set PATH=<values>

[Czaki]

set PY
PyCharm=C:\Program Files\JetBrains\PyCharm 2019.1.2\bin;

set PATH
Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW

And still same error

\pypy3.6-v7.3.0-win32>pypy3.exe
ModuleNotFoundError: No module named 'encodings'
debug: OperationError:
debug:  operror-type: ModuleNotFoundError
debug:  operror-value: No module named 'os'

[Czaki]

@joerick What is status of this PR?

[joerick]

just a couple edits to go, then we can think about merging :)

[joerick]

Alright, I think this is ready! Any more comments before we get this in?

[Czaki]

Upgrade to newest version of package (eg. setuptools)? Or just before release?

[joerick]

Yes, I'll do that before release.

[YannickJadoul]

It's a lot to go through all changes once more, but I guess this is good to be merged? :-)

One thing for a future release: I think PyPy will release a new version (7.3.1) soon. Should we wait for that, or is that for a next release?

[Czaki]

One thing for a future release: I think PyPy will release a new version (7.3.1) soon. Should we wait for that, or is that for a next release?

Next release?

[YannickJadoul]

PyPy 7.3.1 is planned to be released this week (if everything goes well).
If everything goes well for us, a PR replacing 7.3.0 with 7.3.1 should be painless (and hopefully remove some of the current workarounds/patches). So it might be nice to still include that in cibuildwheel 1.4.0?

[Czaki]

PyPy 7.3.1 is planned to be released this week (if everything goes well).
If everything goes well for us, a PR replacing 7.3.0 with 7.3.1 should be painless (and hopefully remove some of the current workarounds/patches). So it might be nice to still include that in cibuildwheel 1.4.0?

If it is painless then yes. If it produce next problem then maybe not wait.

[joerick]

Okay, I'm doing it!

[joerick]

Thanks all for your help with this!

[joerick]

Released as v1.4.0.