GH-45039: [CI][Packaging][Python] Fix Docker push step for free-threaded wheel builds #45040

pitrou · 2024-12-17T10:36:08Z

GitHub Issue: [CI][Packaging][Python] 3.13 free-threaded wheel builds fail in upload step #45039

…-threaded wheel builds

github-actions · 2024-12-17T10:36:38Z

⚠️ GitHub issue #45039 has been automatically assigned in GitHub to PR creator.

pitrou · 2024-12-17T10:37:23Z

@github-actions crossbow submit wheel-manylinux-2-28-cp313-cp313-amd64 wheel-manylinux-2-28-cp313-cp313t-amd64

github-actions · 2024-12-17T10:39:43Z

Revision: 3b6759e

Submitted crossbow builds: ursacomputing/crossbow @ actions-7af60c9692

Task	Status
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313t-amd64

pitrou · 2024-12-17T13:04:44Z

@kou @raulcd Does this seem ok?

pitrou · 2024-12-17T13:06:12Z

@github-actions crossbow submit wheelmanylinux

github-actions · 2024-12-17T13:09:02Z

Revision: 9e6cdb8

Submitted crossbow builds: ursacomputing/crossbow @ actions-552bdb3274

Task	Status
wheel-manylinux-2-28-cp310-cp310-amd64
wheel-manylinux-2-28-cp310-cp310-arm64
wheel-manylinux-2-28-cp311-cp311-amd64
wheel-manylinux-2-28-cp311-cp311-arm64
wheel-manylinux-2-28-cp312-cp312-amd64
wheel-manylinux-2-28-cp312-cp312-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-manylinux-2-28-cp313-cp313t-amd64
wheel-manylinux-2-28-cp313-cp313t-arm64
wheel-manylinux-2-28-cp39-cp39-amd64
wheel-manylinux-2-28-cp39-cp39-arm64
wheel-manylinux-2014-cp310-cp310-amd64
wheel-manylinux-2014-cp310-cp310-arm64
wheel-manylinux-2014-cp311-cp311-amd64
wheel-manylinux-2014-cp311-cp311-arm64
wheel-manylinux-2014-cp312-cp312-amd64
wheel-manylinux-2014-cp312-cp312-arm64
wheel-manylinux-2014-cp313-cp313-amd64
wheel-manylinux-2014-cp313-cp313-arm64
wheel-manylinux-2014-cp313-cp313t-amd64
wheel-manylinux-2014-cp313-cp313t-arm64
wheel-manylinux-2014-cp39-cp39-amd64
wheel-manylinux-2014-cp39-cp39-arm64

raulcd

Thanks @pitrou , LGTM, I am happy to merge without actually pushing the image and just validate on the nightlies

dev/tasks/python-wheels/github.linux.yml

pitrou · 2024-12-17T16:54:31Z

@github-actions crossbow submit wheel-manylinux-2-28-cp313-cp313-amd64 wheel-manylinux-2-28-cp313-cp313t-amd64 wheel-manylinux-2014-cp312-cp312-amd64 wheel-manylinux-2014-cp313-cp313-arm64

github-actions · 2024-12-17T16:56:54Z

Revision: 3f3a068

Submitted crossbow builds: ursacomputing/crossbow @ actions-605028a7bb

Task	Status
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313t-amd64
wheel-manylinux-2014-cp312-cp312-amd64
wheel-manylinux-2014-cp313-cp313-arm64

kou · 2024-12-17T20:46:38Z

dev/tasks/python-wheels/github.linux.yml

        shell: bash
        run: |
          archery docker push python-wheel-manylinux-{{ manylinux_version }}
-          archery docker push python-wheel-manylinux-test-unittests
+          archery docker push {{ test_unittests_image }}


We may want to push test_unittests_image too.

You mean test_imports_image? It's actually an upstream Python image.

python-wheel-manylinux-test-imports: image: ${ARCH}/python:${PYTHON_IMAGE_TAG}

Oh, sorry. Yes. I meant test_imports_image.
I didn't notice that python-wheel-manylinux-test-imports is python:XXX.
I just checked python-free-threaded-wheel-manylinux-test-imports:

arrow/docker-compose.yml

Lines 1207 to 1210 in 37eb3b5

# TODO: Remove this when the official Docker Python image supports the free-threaded build.

# See https://github.com/docker-library/python/issues/947 for more info.

python-free-threaded-wheel-manylinux-test-imports:

image: ${REPO}:${ARCH}-python-3.13-free-threaded-wheel-manylinux-test-imports

We don't need to cache it because it's a temporary image?

Hmm, I don't know. But it's probably not very important, since the build step took 1 minute in https://github.com/ursacomputing/crossbow/actions/runs/12377700603/job/34547835928

OK. Let's keep the current configuration. (We don't cache the python-free-threaded-wheel-manylinux-test-imports image.)

kou · 2024-12-17T20:51:45Z

dev/tasks/python-wheels/github.linux.yml

+# Testing free-threaded wheels uses a different Docker setup
+{% set test_imports_image = (
+    'python-free-threaded-wheel-manylinux-test-imports' if python_abi_tag == 'cp313t'
+    else 'python-wheel-manylinux-test-imports')
+%}
+{% set test_unittests_image = (
+    'python-free-threaded-wheel-manylinux-test-unittests' if python_abi_tag == 'cp313t'
+    else 'python-wheel-manylinux-test-unittests')
+%}
+


I think that using GitHub Actions features instead of using Jinja2 is easier to read/understand:

diff --git a/dev/tasks/python-wheels/github.linux.yml b/dev/tasks/python-wheels/github.linux.yml index f083b7c0c8..eadb948339 100644 --- a/dev/tasks/python-wheels/github.linux.yml +++ b/dev/tasks/python-wheels/github.linux.yml @@ -50,6 +50,15 @@ jobs: {{ macros.github_install_archery()|indent }} {{ macros.github_login_dockerhub()|indent }} + - name: Prepare + run: | + if [ "${PYTHON_ABI_TAG}" = "cp313t" ]; then + test_image_prefix=python-free-threaded + else + test_image_prefix=python + fi + echo "TEST_IMAGE_PREFIX=${test_image_prefix}" >> ${GITHUB_ENV} + - name: Build wheel shell: bash env: @@ -72,23 +81,11 @@ jobs: # TODO(kszucs): auditwheel show - name: Test wheel - if: | - '{{ python_abi_tag }}' != 'cp313t' - shell: bash - run: | - source arrow/ci/scripts/util_enable_core_dumps.sh - archery docker run python-wheel-manylinux-test-imports - archery docker run python-wheel-manylinux-test-unittests - - # Free-threaded wheels need to be tested using a different Docker Compose service - - name: Test free-threaded wheel - if: | - '{{ python_abi_tag }}' == 'cp313t' shell: bash run: | source arrow/ci/scripts/util_enable_core_dumps.sh - archery docker run python-free-threaded-wheel-manylinux-test-imports - archery docker run python-free-threaded-wheel-manylinux-test-unittests + archery docker run ${TEST_IMAGE_PREFIX}-wheel-manylinux-test-imports + archery docker run ${TEST_IMAGE_PREFIX}-wheel-manylinux-test-unittests - name: Test wheel on AlmaLinux 8 shell: bash @@ -145,5 +142,5 @@ jobs: shell: bash run: | archery docker push python-wheel-manylinux-{{ manylinux_version }} - archery docker push python-wheel-manylinux-test-unittests + archery docker push ${TEST_IMAGE_PREFIX}-wheel-manylinux-test-unittests {% endif %}

Hmm, I don't know, both are uncommon to me :) Feel free to open a fixup PR if you want.

apacheGH-45039: [CI][Packaging][Python] Fix Docker push step for free…

3b6759e

…-threaded wheel builds

github-actions bot added the awaiting review Awaiting review label Dec 17, 2024

pitrou marked this pull request as ready for review December 17, 2024 13:04

pitrou requested review from assignUser, jonkeane, kou and raulcd as code owners December 17, 2024 13:04

Fix step name

9e6cdb8

raulcd approved these changes Dec 17, 2024

View reviewed changes

dev/tasks/python-wheels/github.linux.yml Show resolved Hide resolved

github-actions bot added awaiting merge Awaiting merge awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting review Awaiting review awaiting merge Awaiting merge awaiting changes Awaiting changes labels Dec 17, 2024

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Dec 17, 2024

pitrou force-pushed the gh45039-wheel-push branch from 3f3a068 to 9e6cdb8 Compare December 17, 2024 17:21

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 17, 2024

pitrou merged commit 8f336c4 into apache:main Dec 17, 2024
18 checks passed

pitrou removed the awaiting change review Awaiting change review label Dec 17, 2024

pitrou mentioned this pull request Dec 17, 2024

[CI][Packaging][Python] 3.13 free-threaded wheel builds fail in upload step #45039

Closed

pitrou deleted the gh45039-wheel-push branch December 17, 2024 17:24

kou reviewed Dec 17, 2024

View reviewed changes

github-actions bot added the awaiting changes Awaiting changes label Dec 17, 2024

kou mentioned this pull request Dec 19, 2024

[CI][Packaging][Python] Simplify dev/tasks/python-wheels/github.linux.yml #45076

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-45039: [CI][Packaging][Python] Fix Docker push step for free-threaded wheel builds #45040

GH-45039: [CI][Packaging][Python] Fix Docker push step for free-threaded wheel builds #45040

pitrou commented Dec 17, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Dec 17, 2024

pitrou commented Dec 17, 2024

github-actions bot commented Dec 17, 2024

pitrou commented Dec 17, 2024

pitrou commented Dec 17, 2024

github-actions bot commented Dec 17, 2024

raulcd left a comment

pitrou commented Dec 17, 2024

github-actions bot commented Dec 17, 2024

kou Dec 17, 2024

pitrou Dec 18, 2024

kou Dec 18, 2024

pitrou Dec 18, 2024

kou Dec 19, 2024

kou Dec 17, 2024

pitrou Dec 18, 2024

kou Dec 19, 2024

	# TODO: Remove this when the official Docker Python image supports the free-threaded build.
	# See https://github.com/docker-library/python/issues/947 for more info.
	python-free-threaded-wheel-manylinux-test-imports:
	image: ${REPO}:${ARCH}-python-3.13-free-threaded-wheel-manylinux-test-imports

GH-45039: [CI][Packaging][Python] Fix Docker push step for free-threaded wheel builds #45040

GH-45039: [CI][Packaging][Python] Fix Docker push step for free-threaded wheel builds #45040

Conversation

pitrou commented Dec 17, 2024 • edited by github-actions bot Loading

github-actions bot commented Dec 17, 2024

pitrou commented Dec 17, 2024

github-actions bot commented Dec 17, 2024

pitrou commented Dec 17, 2024

pitrou commented Dec 17, 2024

github-actions bot commented Dec 17, 2024

raulcd left a comment

Choose a reason for hiding this comment

pitrou commented Dec 17, 2024

github-actions bot commented Dec 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pitrou commented Dec 17, 2024 •

edited by github-actions bot

Loading