Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish Python 3.13 wheels on PyPI #178

Open
mayeut opened this issue Sep 30, 2023 · 23 comments
Open

Publish Python 3.13 wheels on PyPI #178

mayeut opened this issue Sep 30, 2023 · 23 comments
Assignees
Labels
priority: p3 Desirable enhancement or fix. May not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@mayeut
Copy link
Contributor

mayeut commented Sep 30, 2023

Is your feature request related to a problem? Please describe.
For now, installing python-crc32 with Python 3.12 is done by building from sources rather than just installing a wheel.

Describe the solution you'd like
Please publish Python 3.12 wheels.

Describe alternatives you've considered
Building from sources: this is slow & error prone.

Additional context
Python 3.12 will be released next week and has been ABI stable since rc1.

@mayeut mayeut added priority: p3 Desirable enhancement or fix. May not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. labels Sep 30, 2023
@parthea
Copy link
Contributor

parthea commented Nov 16, 2023

Blocked by #149

@bricker
Copy link

bricker commented Jan 13, 2024

@parthea how can I help get this done?

@martin-traverse
Copy link

martin-traverse commented Jan 23, 2024

We have a multi-cloud product and need this to support GCP with Python 3.12. The current situation is that Python 3.12 is supported on AWS, but not on GCP or Azure because of dependency issues. If there is anything I can do to help, please let me know.

@dsotirho-ucsc
Copy link

This is a blocking us from upgrading to Python 3.12. Any news on this issue would be most appreciated. Thanks!

@ADR-007
Copy link

ADR-007 commented Mar 16, 2024

As a workaround you can build it manually in Docker using this instruction (+several hours for debugging 😀):

ENV PY_BIN=$VIRTUAL_ENV/bin/python
RUN git clone --recursive --depth 1 --branch v1.5.0 https://github.com/googleapis/python-crc32c \
    && cd python-crc32c  \
    && pip install -r scripts/dev-requirements.txt \
    && ./scripts/local-linux/build_libcrc32c.sh \
    && pip install --no-index --find-links=wheels google-crc32c \
    && python ./scripts/check_crc32c_extension.py \
    && python -c "from google_crc32c import *" \
    && ls --hide=usr | xargs -d '\n' rm -rf

@edgarrmondragon
Copy link
Contributor

As a workaround you can build it manually in Docker using this instruction (+several hours for debugging 😀):

ENV PY_BIN=$VIRTUAL_ENV/bin/python
RUN git clone --recursive --depth 1 --branch v1.5.0 https://github.com/googleapis/python-crc32c \
    && cd python-crc32c  \
    && pip install -r scripts/dev-requirements.txt \
    && ./scripts/local-linux/build_libcrc32c.sh \
    && pip install --no-index --find-links=wheels google-crc32c \
    && python ./scripts/check_crc32c_extension.py \
    && python -c "from google_crc32c import *" \
    && ls --hide=usr | xargs -d '\n' rm -rf

Not really a workaround when the request is specifically to avoid having to build from source 😉

@martin-traverse
Copy link

As a workaround you can build it manually in Docker using this instruction (+several hours for debugging 😀):


ENV PY_BIN=$VIRTUAL_ENV/bin/python

RUN git clone --recursive --depth 1 --branch v1.5.0 https://github.com/googleapis/python-crc32c \

    && cd python-crc32c  \

    && pip install -r scripts/dev-requirements.txt \

    && ./scripts/local-linux/build_libcrc32c.sh \

    && pip install --no-index --find-links=wheels google-crc32c \

    && python ./scripts/check_crc32c_extension.py \

    && python -c "from google_crc32c import *" \

    && ls --hide=usr | xargs -d '\n' rm -rf

I do appreciate you trying to offer a solution, but honestly this is a really bad developer experience. If it was just me I wouldn't mind, but our users are modellers, they install our library using pip and expect it to work. We can't start publishing instructions telling them to install Docker and build stuff from source, this is exactly the kind of pain we are trying to solve for them. Also the tools available in enterprise environments are normally hugely restricted, if Docker is even an option you probably have to raise a bunch of requests and go through an approval workflow which can take an unlimited amount of time, maybe you even need an entirely different machine and have to set up all your tools again. It's just not practical and our users won't do it.

We will have to continue telling our users that GCP doesn't support Python 3.12 until the packages are available through the normal mechanism. This is what we need:

pip install google-auth
pip install google-cloud-core
pip install google-cloud-storage

I hope this helps explain the context a little bit! In my field there are lots of people who want to use these tools but issues like this will stop them in their tracks.

@edgarrmondragon
Copy link
Contributor

Gentle bump. Python 3.12 is now 6 months old.

@parthea parthea assigned andrewsg and unassigned parthea Apr 4, 2024
@parthea
Copy link
Contributor

parthea commented Apr 4, 2024

See #149 (comment) for ETA

@jlaine
Copy link

jlaine commented May 20, 2024

Could you conceivably make use of Python's limited API [1]? This would avoid having to rebuild wheels whenever a new version of Python is released.

[1] https://docs.python.org/3/c-api/stable.html#limited-c-api

@hannes-ucsc
Copy link

Why is #149 a blocker?

This appears to be affecting everyone attempting to use the Google Cloud SDK with Python 3.12, and should therefore be expedited, no?

@hannes-ucsc
Copy link

hannes-ucsc commented May 29, 2024

@ADR-007

As a workaround you can build it manually in Docker …

Thank you!

@adamchainz
Copy link

To put some practical numbers on the message “significantly slower”, I ran a mini benchmark, checksumming 1MiB of data on both versions.

Python 3.11, with extension:

In [1]: import google_crc32c

In [2]: %timeit cs = google_crc32c.Checksum() ; cs.update(b'1' * 1024 * 1024) ; cs.digest()
244 µs ± 4.62 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Python 3.12, without extension:

In [1]: import google_crc32c
/.../google_crc32c/__init__.py:29: RuntimeWarning: As the c extension couldn't be imported, `google-crc32c` is using a pure python implementation that is significantly slower. If possible, please configure a c build environment and compile the extension
  warnings.warn(_SLOW_CRC32C_WARNING, RuntimeWarning)

In [2]: %timeit cs = google_crc32c.Checksum() ; cs.update(b'1' * 1024 * 1024) ; cs.digest()
125 ms ± 1.63 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

It’s about ~500,000 times slower. Perhaps one could budget an extra 100ms for each Google API request.

This degradation is not tolerable for the project I’m working on, so we’ll have to delay upgrading Python or workaround and compile ourselves.

@hannes-ucsc
Copy link

This bash command builds the wheels and drops them in the current directory:

for arch in arm64 amd64
do docker run --rm -iv "$PWD":/host --platform=linux/$arch python:3.12 /bin/bash <<-"END"
    set -ex
    export PY_BIN="$(which python)"
    git clone --recursive --depth 1 --branch v1.5.0 https://github.com/googleapis/python-crc32c
    cd python-crc32c
    pip install -r scripts/dev-requirements.txt
    ./scripts/local-linux/build_libcrc32c.sh
    pip install --no-index --find-links=wheels google-crc32c
    python ./scripts/check_crc32c_extension.py
    python -c "from google_crc32c import *"
    cp wheels/*.whl /host
END
done

Based on #178 (comment) by @ADR-007

@andrewsg
Copy link
Contributor

This work is in progress, we're having some build system issues on Windows and Apple Silicon. Hope to have an update soon. Thanks all.

@v-amorim
Copy link

I couldn't solve it for my 3.12.* version, so I supressed the message as such:

import warnings # before any from x

warnings.filterwarnings("ignore", category=RuntimeWarning, module="google_crc32c") # before any from x

I added this on my __init__.py, but can be used on any file that will use the library.

@edgarrmondragon
Copy link
Contributor

@v-amorim note that doing that will just silence the warning but the real problem is the performance downgrade exemplified in #178 (comment).

@mayeut mayeut changed the title Publish Python 3.12 wheels on PyPI Publish Python 3.12/3.13 wheels on PyPI Aug 4, 2024
@mayeut
Copy link
Contributor Author

mayeut commented Aug 4, 2024

Python 3.13.0rc1 is ABI stable so publishing 3.13 wheels can also be done.

@akshit0201
Copy link

akshit0201 commented Aug 9, 2024

This issue is a blocker for me, I can't deploy to AWS Lambda.
Don't know why this issue was closed when the wheel isn't there on PyPI: https://pypi.org/project/google-crc32c/#files

@adamchainz
Copy link

Don't know why this issue was closed

It’s not closed.

@ondrejdavidod
Copy link

Hi, since the blocker #149 for this task is merged already, when can we expect the fix for this one, please? 🙏 Thank you.

@parthea parthea changed the title Publish Python 3.12/3.13 wheels on PyPI Publish Python 3.13 wheels on PyPI Sep 3, 2024
@parthea
Copy link
Contributor

parthea commented Sep 3, 2024

@ondrejdavidod, the normal release for Python 3.12 will be available today. For Python 3.13, we're still working on it.

@edgarrmondragon
Copy link
Contributor

There's a PR that adds support for building cp313 wheels: #254

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p3 Desirable enhancement or fix. May not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests