Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update pants release to support py2.py3. #6450

Closed
jsirois opened this issue Sep 4, 2018 · 5 comments
Closed

Update pants release to support py2.py3. #6450

jsirois opened this issue Sep 4, 2018 · 5 comments

Comments

@jsirois
Copy link
Contributor

jsirois commented Sep 4, 2018

We'll need to update at least 3 items in tandem:

  1. pantsbuild.pants will need something like --python-tag cp27.cp34.cp35.cp36.cp37

    pants/build-support/bin/release.sh

    Lines 63 to 68 in 1ec7de8

    PKG_PANTS=(
    "pantsbuild.pants"
    "//src/python/pants:pants-packaged"
    "pkg_pants_install_test"
    "--python-tag cp27 --plat-name $(find_plat_name)"
    )
  2. contrib plugins will need something like --python-tag py27.py34.py35.py36.py37:

    pants/build-support/bin/release.sh

    Lines 214 to 215 in 1ec7de8

    run_local_pants setup-py \
    --run="bdist_wheel ${BDIST_WHEEL_FLAGS:---python-tag py27}" \
  3. The pants_requirement environment markers will need to be updated to include the same set of python versions as implied by the --python-tags above:

    pants/src/python/pants/backend/python/pants_requirement.py

    Line 53 in 1ec7de8

    env_marker = "python_version>='3.4' or (python_version>='2.7' and python_version<'3')"
@jsirois
Copy link
Contributor Author

jsirois commented Sep 4, 2018

Part of the work to close #6062

@jsirois
Copy link
Contributor Author

jsirois commented Sep 4, 2018

Not updating the three items above led to a bug discovered in #6448

@jsirois
Copy link
Contributor Author

jsirois commented Dec 18, 2018

Noting that items 1 & 2 have moved to:

pants/src/python/pants/releases/packages.py

Line 38 in 6457a75

self.bdist_wheel_flags = bdist_wheel_flags or ("--python-tag", "py27")

pants/src/python/pants/releases/packages.py

Line 95 in 6457a75

bdist_wheel_flags=("--python-tag", "cp27", "--plat-name", find_platform_name()),

@stuhood stuhood mentioned this issue Jan 25, 2019
Closed
@jsirois
Copy link
Contributor Author

jsirois commented Jan 29, 2019 •
edited
Loading

Noting that the pex side of releases is blocked on at least:

  1. The --interpreter-constraint option should OR pex-tool/pex#655
  2. If an --interpreter-constraint is set, it should always be honored. pex-tool/pex#656

These are the same issues pex itself faces in trying to release a multi-python pex of itself as outlined in pex-tool/pex#654.

There is at least one more blocking issue in pex-tool/pex#415 where pex fails to resolve / build for multi-python when the dist in question is not already universal. I'll link that issue as well once filed.

Eric-Arellano added a commit to Eric-Arellano/pants that referenced this issue Jan 31, 2019
@Eric-Arellano
Implement pantsbuild#6450
77d730e 
Although our solution is going to end up releasing a separate wheel for Py2 vs Py3 for pantsbuild.pants, this is a good starting point for the changes we need to make.
Eric-Arellano added a commit to Eric-Arellano/pants that referenced this issue Feb 1, 2019
@Eric-Arellano
Implement pantsbuild#6450
1b556c9 
Although our solution is going to end up releasing a separate wheel for Py2 vs Py3 for pantsbuild.pants, this is a good starting point for the changes we need to make.
@Eric-Arellano Eric-Arellano mentioned this issue Feb 1, 2019
Merged
Eric-Arellano added a commit that referenced this issue Mar 14, 2019
@Eric-Arellano
Add support for releasing Python 3 wheels (#7197)
b3ab38c 
### Problem
The final step in #6062 is to add support for releasing Pants and its contrib packages as compatible with Python 2.7 _and_  3.6 and 3.7. Specifically, universal wheels should be marked as `py27.py36.py37` and `pantsbuild.pants` should be marked as `cp27-cp27{m,mu}` or as `cp36-abi3` because it uses native code (see [comment in packages.py](https://github.com/pantsbuild/pants/pull/7197/files#diff-cccea5c0944f25138aa83f5815236ea2R95) for an explanation on `abi3`).

`packages.py` and `release.sh` had several issues preventing building Python 3 wheels. Further, we needed to add shards to build Python 3 wheels in CI.

This resolves the wheels portion of #6450.

### Solution
* Add a `-3` CLI flag to `release.sh` to indicate it should build wheels compatible with Py3. Also add `--py3` to `packages.py`.
* Update the default `Package` config to setup support for `py27.py36.py37`, meaning any Python implementation that is one of those 3 versions.
* Update the `pantsbuild.pants` flags to indicate either `cp27` or `cp36-abi3`. The former is only compatible with CPython2.7, and the latter is compatible with any CPython implementation >= 3.6 due to the ABI constraint (see https://docs.python.org/3/c-api/stable.html).
* Add Python 3 wheels shards to CI.
* Pull down either a pex27 or pex36 release, rather than alway using pex27.
* Modernize Python code, such as fixing bad `subprocess32` import.
* Remove the compatibility tags from `PantsRequirement`. John explained it was wrong to put here, I think because it should be marked in` packages.py` when we actually build the wheels? This results in also tweaking the corresponding unit tests.

### Result
`./build-support/bin/release.sh -3n` builds Python 3 compatible wheels ready for release.

Note we still cannot release a Python 3 compatible PEX, which is blocked by pex-tool/pex#654.
@Eric-Arellano Eric-Arellano mentioned this issue Mar 19, 2019
Merged
Eric-Arellano added a commit that referenced this issue Mar 20, 2019
@Eric-Arellano
Release pants as both a Python 2.7 PEX and a Python 3.6 PEX (#7401)
d392a19 
### Problem
While wheels are now released with Python 3 support thanks to #7197, this is only one of two ways people consume Pants. The other way is through a Pex released to Github. 

We decided to release both a Py27 and Py36 Pex. 

We will not be releasing a Py37 Pex, as it is would require Py37 wheel building shards which would require a Centos7 docker image. We decided this is not worth the work for how few people we anticipate would use Py37 with Pex combined.

We will also not be releasing a more flexible Pex that works with either Py27 or Py36, as it is blocked by pex-tool/pex#654. We also want organizations pinning their Python version, so a flexible release is not particularly useful in this instance.

This implements #6450 (comment).

### Solution
Release both `pants.${VERSION}.py{27,36}.pex`.

This requires changing how we run `release.sh -p` and adding new CI shards to deploy on both Py2 and Py3.

### Result
The next release, we will release both `pants.1.15.0.rc1.py27.pex` and  `pants.1.15.0.rc1.py36.pex`.
stuhood pushed a commit that referenced this issue Mar 21, 2019
@Eric-Arellano @stuhood
Release pants as both a Python 2.7 PEX and a Python 3.6 PEX (#7401)
b78a4d2 
### Problem
While wheels are now released with Python 3 support thanks to #7197, this is only one of two ways people consume Pants. The other way is through a Pex released to Github. 

We decided to release both a Py27 and Py36 Pex. 

We will not be releasing a Py37 Pex, as it is would require Py37 wheel building shards which would require a Centos7 docker image. We decided this is not worth the work for how few people we anticipate would use Py37 with Pex combined.

We will also not be releasing a more flexible Pex that works with either Py27 or Py36, as it is blocked by pex-tool/pex#654. We also want organizations pinning their Python version, so a flexible release is not particularly useful in this instance.

This implements #6450 (comment).

### Solution
Release both `pants.${VERSION}.py{27,36}.pex`.

This requires changing how we run `release.sh -p` and adding new CI shards to deploy on both Py2 and Py3.

### Result
The next release, we will release both `pants.1.15.0.rc1.py27.pex` and  `pants.1.15.0.rc1.py36.pex`.
@Eric-Arellano
Copy link
Contributor

This is closed by #7197 and #7401. Our first Pex release will be 1.15.0.rc1.

jsirois added a commit to jsirois/pants that referenced this issue Apr 15, 2019
@jsirois
Directory materialization is visible to others.
db204ba 
Previously when the store materialized directories it would not fsync.
This led to the possibility that processes spawned against chroots
materialized by the store would see only partial files.

Fixes pantsbuild#6450
@jsirois jsirois mentioned this issue Apr 15, 2019
Merged
jsirois added a commit that referenced this issue Apr 15, 2019
@jsirois
Directory materialization is visible to others. (#7562)
370e650 
Previously when the store materialized directories it would not fsync.
This led to the possibility that processes spawned against chroots
materialized by the store would see only partial files.

Fixes #6450
@asherf asherf mentioned this issue Jan 12, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jsirois @Eric-Arellano