Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deterministic builds #256

Merged
merged 74 commits into from
Apr 5, 2020
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
06696c4
Pin tool dependencies whenever we pip install using a constraints file
joerick Jan 28, 2020
1cae983
Add docs for CIBW_DEPENDENCY_VERSIONS
joerick Jan 28, 2020
c0967ad
Fix docker copy
joerick Jan 28, 2020
b159617
Vendor get-pip (version 19.3.1)
joerick Jan 31, 2020
042efd6
Update constraints
joerick Jan 31, 2020
1b9b6e9
Add the ability to pin docker image tags (this isn't consumed yet)
joerick Jan 31, 2020
8ead985
Generate separate python2 constraint files
joerick Jan 31, 2020
bad72cb
Merge remote-tracking branch 'origin/master' into deterministic-builds
joerick Feb 2, 2020
076987c
Move the config flow control of the linux build into Python
joerick Feb 2, 2020
b9607ff
Add ability to pin different dependencies for each python version
joerick Feb 2, 2020
baa0897
Add unit test case and fix filename confusion
joerick Feb 2, 2020
6177408
Consume our pinned manylinux image version file for the defaults
joerick Feb 2, 2020
e32d0af
Allow any version of the docker image in the unit test
joerick Feb 2, 2020
d8a245d
Use abspath for constraints file
joerick Feb 2, 2020
a953c5f
Remove auditwheel from pinned deps - it's preinstalled on manylinux
joerick Feb 3, 2020
bf65679
Fix imports on Python 2
joerick Feb 3, 2020
17a019b
Neaten comment
joerick Feb 3, 2020
ada46aa
Fix exports / version check ordering
joerick Feb 3, 2020
1033817
Merge remote-tracking branch 'origin/master' into deterministic-builds
joerick Feb 15, 2020
f2c2a06
Merge fixes and code style
joerick Feb 15, 2020
50bd6f4
Merge branch 'master' into deterministic-builds
joerick Feb 15, 2020
4c076b8
Fix piping script into docker bash
joerick Feb 15, 2020
bf87df5
Ensure subprocess.run commands succeed with check=True
joerick Feb 15, 2020
fe3abfb
Docs updates
joerick Feb 18, 2020
c9e4a66
Add a test case for pinned tool versions
joerick Feb 18, 2020
b7fc45a
Fix expected wheels in test
joerick Feb 18, 2020
3c20f70
Add test for user-specified constraints file
joerick Feb 18, 2020
ddc4fb2
Add tests for pinned software in setup.py of test project
joerick Feb 18, 2020
fd69aed
Remove dead code
joerick Feb 18, 2020
f55a5d4
Fix typo and add debug logging
joerick Feb 18, 2020
fc20273
Remove the unit test check - the versions are determined by virtualen…
joerick Feb 18, 2020
5187633
Add the --no-download flag to virtualenv, to prevent it getting the l…
joerick Feb 18, 2020
8a450b7
Fix expected wheel test logic
joerick Feb 18, 2020
5e16e86
Fix some comments from review
joerick Mar 1, 2020
4c7453d
Neater test implementation based on review
joerick Mar 1, 2020
1880b6d
Merge remote-tracking branch 'origin/master' into deterministic-builds
joerick Mar 1, 2020
9768897
Add other architectures and pypy to pinned image config.
joerick Mar 7, 2020
7e583dc
Update constraints
joerick Mar 7, 2020
298df76
Consume new config file with pypy and extra architectures
joerick Mar 7, 2020
2192662
Fix missing pypy image name
joerick Mar 7, 2020
54fda3f
Add back the version field to manylinux PythonConfiguration
joerick Mar 7, 2020
e06b769
Add missing function call, which causing tests to be skipped
joerick Mar 7, 2020
04531d9
Fix unit tests
joerick Mar 7, 2020
dbc2f06
Fix constraint resolution on pypy-2.7
joerick Mar 7, 2020
7c6e69e
Add separate constraints for Python3.5
joerick Mar 7, 2020
2f1ec58
Add constraints for python3.6 too
joerick Mar 7, 2020
ad1b59f
Remove hashes from constraint files
joerick Mar 7, 2020
fd1cd24
Merge remote-tracking branch 'origin/master' into deterministic-builds
joerick Mar 7, 2020
dd2c27a
Some test fixes related to PyPy
joerick Mar 7, 2020
02a5279
Fix incorrect constraints file on pypy2.7
joerick Mar 8, 2020
7ab2d59
Merge branch 'master' of github.com:joerick/cibuildwheel into determi…
joerick Mar 10, 2020
8603dea
Update constraints
joerick Mar 10, 2020
4fb4e44
Fix some errors introduced after the merge
joerick Mar 10, 2020
cc956ea
Update version of setuptools used to test constraint file
joerick Mar 11, 2020
b55e98f
Update test-pinned version of pip
joerick Mar 11, 2020
b4ed192
Run dependency tests on pypy too, and update constraint for 3.6
joerick Mar 11, 2020
7efec9a
Try updating more things
joerick Mar 11, 2020
c4f54c6
Run each build using `docker exec`, for faster builds
joerick Mar 11, 2020
e49644e
Some docker fixes
joerick Mar 12, 2020
4e31c65
Workaround `docker cp` bug
joerick Mar 13, 2020
94824f9
Merge remote-tracking branch 'origin/master' into deterministic-builds
joerick Mar 13, 2020
2db9d9a
The docker container needs a main process
joerick Mar 13, 2020
7090bfa
Stop isn't necessary, `docker rm` will stop the container
joerick Mar 13, 2020
580b065
Add no_output_timeout to CircleCI tests to prevent them timing out
joerick Mar 13, 2020
dc5dbea
See stdout during tests to debug circleCI
joerick Mar 14, 2020
7316d69
Improve logging of docker calls
joerick Mar 14, 2020
98dbbae
Fix docker build on CircleCI
joerick Mar 14, 2020
36a2f00
Revert "See stdout during tests to debug circleCI"
joerick Mar 14, 2020
33da3d6
Fix CUSTOM_COMPILE_COMMAND and add comment
joerick Mar 17, 2020
ffaad91
Remove PyPy version from PythonConfiguration
joerick Mar 18, 2020
de4bf6d
Add missing version-specific constraint tests
joerick Mar 23, 2020
4b6df45
Docs wording edit
joerick Mar 23, 2020
13554d0
Call get-pip.py on Windows with dependency constraint flags
joerick Mar 24, 2020
3fa9935
Ensure the correct versions of constraint files are used for each ver…
joerick Mar 28, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -100,3 +100,4 @@ env2/

# VSCode project settings
/.vscode
env2
joerick marked this conversation as resolved.
Show resolved Hide resolved
4 changes: 4 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,7 @@ SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


Get-pip and Pip (bundled in cibuildwheel/resources) are licensed under the MIT
license. See https://github.com/pypa/get-pip/blob/master/LICENSE.txt
59 changes: 59 additions & 0 deletions bin/update_constraints
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#!/usr/bin/env python3

import os, subprocess, requests, configparser
from collections import namedtuple

os.chdir(os.path.dirname(__file__))
os.chdir('..')

os.environ['CUSTOM_COMPILE_COMMAND'] = "bin/update_constraints"
subprocess.check_call([
'pip-compile',
'--allow-unsafe',
'--generate-hashes',
'--upgrade',
'cibuildwheel/resources/constraints.in',
])
subprocess.check_call([
'./env2/bin/pip-compile',
'--allow-unsafe',
'--generate-hashes',
'--upgrade',
'cibuildwheel/resources/constraints.in',
'--output-file', 'cibuildwheel/resources/constraints-python27.txt'
])

Image = namedtuple('Image', ['manylinux_version', 'arch'])

images = [
Image('manylinux1', 'x86_64'), Image('manylinux1', 'i686'),
Image('manylinux2010', 'x86_64'), Image('manylinux2010', 'i686'),
Image('manylinux2014', 'x86_64'), Image('manylinux2014', 'i686'),
joerick marked this conversation as resolved.
Show resolved Hide resolved
]

config = configparser.ConfigParser()

for image in images:
repository_name = '{}_{}'.format(image.manylinux_version, image.arch)
quay_api_url = 'https://quay.io/api/v1/repository/pypa/{}?includeTags=true'.format(repository_name)
repo_info = requests.get(quay_api_url).json()
tags_dict = repo_info['tags']

latest_tag = tags_dict.pop('latest')

# find the tag whose manifest matches 'latest'
tag_name = next(
name
for (name, info) in tags_dict.items()
if info['manifest_digest'] == latest_tag['manifest_digest']
)

image_name = 'quay.io/pypa/{}_{}:{}'.format(image.manylinux_version, image.arch, tag_name)

if not config.has_section(image.manylinux_version):
config[image.manylinux_version] = {}

config[image.manylinux_version][image.arch] = image_name

with open('cibuildwheel/resources/pinned_docker_images.cfg', 'w') as f:
config.write(f)
YannickJadoul marked this conversation as resolved.
Show resolved Hide resolved
42 changes: 34 additions & 8 deletions cibuildwheel/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import sys
import textwrap
import traceback
from configparser import ConfigParser

import cibuildwheel
import cibuildwheel.linux
Expand All @@ -14,6 +15,7 @@
)
from cibuildwheel.util import (
BuildSelector,
DependencyConstraints,
Unbuffered
)

Expand Down Expand Up @@ -114,6 +116,14 @@ def main():
repair_command = get_option_from_environment('CIBW_REPAIR_WHEEL_COMMAND', platform=platform, default=repair_command_default)
environment_config = get_option_from_environment('CIBW_ENVIRONMENT', platform=platform, default='')

dependency_versions = get_option_from_environment('CIBW_DEPENDENCY_VERSIONS', platform=platform, default='pinned')
if dependency_versions == 'pinned':
dependency_constraints = DependencyConstraints.with_defaults()
elif dependency_versions == 'latest':
dependency_constraints = None
else:
dependency_constraints = DependencyConstraints(dependency_versions)

if test_extras:
test_extras = '[{0}]'.format(test_extras)

Expand Down Expand Up @@ -154,22 +164,38 @@ def main():
build_selector=build_selector,
repair_command=repair_command,
environment=environment,
dependency_constraints=dependency_constraints,
)

if platform == 'linux':
manylinux_x86_64_image = os.environ.get('CIBW_MANYLINUX_X86_64_IMAGE', 'manylinux2010')
manylinux_i686_image = os.environ.get('CIBW_MANYLINUX_I686_IMAGE', 'manylinux2010')

default_manylinux_images_x86_64 = {'manylinux1': 'quay.io/pypa/manylinux1_x86_64',
'manylinux2010': 'quay.io/pypa/manylinux2010_x86_64',
'manylinux2014': 'quay.io/pypa/manylinux2014_x86_64'}
default_manylinux_images_i686 = {'manylinux1': 'quay.io/pypa/manylinux1_i686',
'manylinux2010': 'quay.io/pypa/manylinux2010_i686',
'manylinux2014': 'quay.io/pypa/manylinux2014_i686'}
pinned_docker_images_file = os.path.join(
os.path.dirname(__file__), 'resources', 'pinned_docker_images.cfg'
YannickJadoul marked this conversation as resolved.
Show resolved Hide resolved
)
pinned_docker_images = ConfigParser()
pinned_docker_images.read(pinned_docker_images_file)
joerick marked this conversation as resolved.
Show resolved Hide resolved

# pinned_docker_images looks like a dict of dicts, e.g.
joerick marked this conversation as resolved.
Show resolved Hide resolved
# { 'manylinux1': {'x86_64': '...', 'i686': '...'},
# 'manylinux2010': {'x86_64': '...', 'i686': '...'},
# 'manylinux2014': {'x86_64': '...', 'i686': '...'} }

manylinux_images = {}

if manylinux_x86_64_image in pinned_docker_images:
joerick marked this conversation as resolved.
Show resolved Hide resolved
manylinux_images['x86_64'] = pinned_docker_images[manylinux_x86_64_image]['x86_64']
else:
manylinux_images['x86_64'] = manylinux_x86_64_image

if manylinux_i686_image in pinned_docker_images:
manylinux_images['i686'] = pinned_docker_images[manylinux_i686_image]['i686']
else:
manylinux_images['i686'] = manylinux_i686_image

build_options.update(
manylinux_images={'x86_64': default_manylinux_images_x86_64.get(manylinux_x86_64_image) or manylinux_x86_64_image,
'i686': default_manylinux_images_i686.get(manylinux_i686_image) or manylinux_i686_image},
manylinux_images=manylinux_images
)
elif platform == 'macos':
pass
Expand Down
Loading