PYTHON-3356 Build pymongocrypt wheels in CI #418

blink1073 · 2022-07-15T14:53:28Z

No description provided.

ShaneHarvey

It looks like the macos framework pythons are broken:

 [2022/07/28 21:39:09.138] running build_ext
 [2022/07/28 21:39:09.138] Running test with python: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3
 [2022/07/28 21:39:09.138] + rm -rf .venv
 [2022/07/28 21:39:09.138] + for PYTHON_BINARY in '"${PYTHONS[@]}"'
 [2022/07/28 21:39:09.138] + echo 'Running test with python: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3'
 [2022/07/28 21:39:09.138] + /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 -c 'import sys; print(sys.version)'
 [2022/07/28 21:39:09.289] 3.8.10 (v3.8.10:3d8993a744, May  3 2021, 09:09:08)
 [2022/07/28 21:39:09.289] [Clang 12.0.5 (clang-1205.0.22.9)]
 [2022/07/28 21:39:09.290] + createvirtualenv /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 .venv
 [2022/07/28 21:39:09.290] + PYTHON=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3
 [2022/07/28 21:39:09.290] + VENVPATH=.venv
 [2022/07/28 21:39:09.290] + /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 -m virtualenv --version
 [2022/07/28 21:39:09.340] /Library/Frameworks/Python.framework/Versions/3.8/bin/python3: No module named virtualenv
 [2022/07/28 21:39:09.470] + /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 -m venv -h
 [2022/07/28 21:39:09.470] Traceback (most recent call last):
 [2022/07/28 21:39:09.470]   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 64, in <module>
 [2022/07/28 21:39:09.470]     import msvcrt
 [2022/07/28 21:39:09.470] ModuleNotFoundError: No module named 'msvcrt'
 [2022/07/28 21:39:09.470] During handling of the above exception, another exception occurred:
 [2022/07/28 21:39:09.470] Traceback (most recent call last):
 [2022/07/28 21:39:09.470]   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 185, in _run_module_as_main
 [2022/07/28 21:39:09.470]     mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
 [2022/07/28 21:39:09.470]   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 144, in _get_module_details
 [2022/07/28 21:39:09.470]     return _get_module_details(pkg_main_name, error)
 [2022/07/28 21:39:09.470]   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 111, in _get_module_details
 [2022/07/28 21:39:09.470]     __import__(pkg_name)
 [2022/07/28 21:39:09.470]   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/venv/__init__.py", line 10, in <module>
 [2022/07/28 21:39:09.470]     import subprocess
 [2022/07/28 21:39:09.470]   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 70, in <module>
 [2022/07/28 21:39:09.470]     import select
 [2022/07/28 21:39:09.470] ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/select.cpython-38-darwin.so, 2): Symbol not found: ___darwin_check_fd_set_overflow
 [2022/07/28 21:39:09.470]   Referenced from: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/select.cpython-38-darwin.so (which was built for Mac OS X 11.0)
 [2022/07/28 21:39:09.470]   Expected in: /usr/lib/libSystem.B.dylib
 [2022/07/28 21:39:09.470]  in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/select.cpython-38-darwin.so
 [2022/07/28 21:39:09.475] + command -v virtualenv
 [2022/07/28 21:39:09.475] ++ command -v virtualenv
 [2022/07/28 21:39:09.475] + VIRTUALENV='/usr/local/bin/virtualenv -p /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 --system-site-packages --never-download'
 [2022/07/28 21:39:09.475] + /usr/local/bin/virtualenv -p /Library/Frameworks/Python.framework/Versions/3.8/bin/python3 --system-site-packages --never-download .venv
 [2022/07/28 21:39:09.966] /usr/local/bin/virtualenv
 [2022/07/28 21:39:09.966] created virtual environment CPython3.8.10.final.0-64 in 358ms
 [2022/07/28 21:39:09.966]   creator CPython3Posix(dest=/data/mci/2a6095fdcfc5eb875c7b80b040ad3c72/libmongocrypt/bindings/python/.venv, clear=False, no_vcs_ignore=False, global=True)
 [2022/07/28 21:39:09.966]   seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/Users/mci/Library/Application Support/virtualenv)
 [2022/07/28 21:39:09.966]     added seed packages: pip==22.1.2, setuptools==62.6.0, wheel==0.37.1
 [2022/07/28 21:39:09.966]   activators NushellActivator,PythonActivator,FishActivator,CShellActivator,PowerShellActivator,BashActivator

https://spruce.mongodb.com/task/libmongocrypt_macos_x86_64_test_python_patch_c2712248e9f4909cdad723607ea5291d2eb48b91_62e2ffb5a4cf470f9c7187c7_22_07_28_21_29_26/logs?execution=0

This might explain why the wheels are intel and not x86_64.

ShaneHarvey · 2022-07-28T22:18:55Z

What if we just produce 2 macos wheels?

macosx_11_0_universal2.whl: with Python 3.10 bundling the universal libmongocrypt.
macosx_10_14_x86_64.whl: with Python 3.7 bundling the x86-64-only libmongocrypt.

These two should give us the maximum compatibility.

blink1073 · 2022-07-28T22:38:17Z

What if we just produce 2 macos wheels?

Sounds good

blink1073 · 2022-08-23T23:37:14Z

We end up with the following in the uploaded tar ball:

ShaneHarvey · 2022-08-24T02:28:32Z

.evergreen/config.yml

+
+- name: "release-python-linux"
+  tags: ["release_python_tag"]
+  run_on: ubuntu2004-large


Can we use ubuntu2004-small instead?

ShaneHarvey · 2022-08-24T02:30:26Z

.evergreen/config.yml

+    - func: "build python release"
+    - func: "test python release"
+      vars: { PYTHON: python }
+    - func: "upload python release"


Can you add the sdist to one of these tasks so that the tar.gz release is included too?

ShaneHarvey · 2022-08-24T02:30:58Z

bindings/python/release.sh

-
-    $PYTHON setup.py bdist_wheel
-    rm -rf build libmongocrypt pymongocrypt/*.so pymongocrypt/*.dll pymongocrypt/*.dylib
+    sudo rm -rf build libmongocrypt pymongocrypt/*.so pymongocrypt/*.dll pymongocrypt/*.dylib


Is this still a problem now that we're using Evergreen?

blink1073 · 2022-08-24T22:01:29Z

Now includes sdist:

blink1073 added 30 commits July 15, 2022 09:50

PYTHON-3356 Build pymongocrypt wheels in CI

be0271c

fix workflow syntax

a644120

trigger build

a0965f0

fixups

dd37234

fixups

41cfaa8

fixups

f40ce57

refactor

88af549

syntax

50d1a5b

syntax

fd94edf

syntax

4149ea3

syntax

172cead

syntax

5e780ea

syntax

8c0a668

syntax

fcc2b6f

fixup

9b7e2cb

try git fetch

ec4b9f9

try again

482f241

cleanup

a6e58c8

cleanup

430fcce

try all tags

3510a39

try again

4267be0

use python setup.py

4fb475b

spelling

031fcd7

no universal

d0dc631

refactor

881110b

fix workflow

f78900a

try again

38e2ebd

update event trigger

13a475e

try again

dc374be

use system python

95cb26b

use explicit windows variant

f6c1ca4

ShaneHarvey reviewed Jul 28, 2022

View reviewed changes

blink1073 added 17 commits July 28, 2022 17:38

clean up wheel building and testing

5788464

fix subs

5e55be5

another windows attempt

0e4cce4

fix variant name

b789045

fix config syntax

255601e

do not use build variant in s3 path

3673939

remove undefined var

240ba4c

move the cp command out of silent part

8f057bf

use auth helper

5b14aab

better handling of creds

33b856b

try without recursive

97fb36d

fix release handling

3fa7c7a

merge from master

cfed8ab

update config

32c622c

fix test python release

4a90209

fix python binary

cd988e6

clean up wheel building

a1b2047

blink1073 requested a review from ShaneHarvey August 23, 2022 23:37

ShaneHarvey requested changes Aug 24, 2022

View reviewed changes

blink1073 added 2 commits August 24, 2022 15:12

address review

db61eb6

cleanup

6e19667

ShaneHarvey approved these changes Aug 24, 2022

View reviewed changes

blink1073 merged commit e6e9054 into mongodb:master Aug 25, 2022

blink1073 deleted the PYTHON-3356 branch August 25, 2022 02:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PYTHON-3356 Build pymongocrypt wheels in CI #418

PYTHON-3356 Build pymongocrypt wheels in CI #418

blink1073 commented Jul 15, 2022

ShaneHarvey left a comment

ShaneHarvey commented Jul 28, 2022 •

edited

Loading

blink1073 commented Jul 28, 2022

blink1073 commented Aug 23, 2022

ShaneHarvey Aug 24, 2022

blink1073 Aug 24, 2022

ShaneHarvey Aug 24, 2022

blink1073 Aug 24, 2022

ShaneHarvey Aug 24, 2022

blink1073 commented Aug 24, 2022

PYTHON-3356 Build pymongocrypt wheels in CI #418

PYTHON-3356 Build pymongocrypt wheels in CI #418

Conversation

blink1073 commented Jul 15, 2022

ShaneHarvey left a comment

Choose a reason for hiding this comment

ShaneHarvey commented Jul 28, 2022 • edited Loading

blink1073 commented Jul 28, 2022

blink1073 commented Aug 23, 2022

ShaneHarvey Aug 24, 2022

Choose a reason for hiding this comment

blink1073 Aug 24, 2022

Choose a reason for hiding this comment

ShaneHarvey Aug 24, 2022

Choose a reason for hiding this comment

blink1073 Aug 24, 2022

Choose a reason for hiding this comment

ShaneHarvey Aug 24, 2022

Choose a reason for hiding this comment

blink1073 commented Aug 24, 2022

ShaneHarvey commented Jul 28, 2022 •

edited

Loading