gh-132983: Introduce `_zstd` bindings module #133027

emmatyping · 2025-04-26T19:38:51Z

This is part 2 (but parallel to part 1) for implementing PEP 784. This is probably the meat of the implementation making up ~half of the LOC in the overall diff.

This change:

Adds the pyzstd license, since this is the first code introduced based on pyzstd
Adds the code for the _zstd module in Modules/_zstd. Tests will be included when the Python compression.zstd module is added. If people think the tests should be included, I can merge the compression.zstd changes into the PR, but I wanted to separate them to make the PRs a bit more manageable.
Adds unix build changes (Windows requires changes in cpython-source-deps and will be added in a follow-up PR)

I added skip news as I'd like to write a holistic NEWS/What's New entry once the entire implementation has landed. If people think each PR should have NEWS I can write something up.

TODO before merge:

try running tests on top of this branch
format PEP 7 style
run buildbots on this PR
verify _zstd gets installed

Issue: Implement PEP 784 - Adding Zstandard to the Python standard library #132983

📚 Documentation preview 📚: https://cpython-previews--133027.org.readthedocs.build/

emmatyping · 2025-04-26T19:47:42Z

~~Hm, I'll dig into the C global failure. Didn't show up when the tests ran on the branch :(~~

E: now fixed

ZeroIntensity

Nice! Some comments at a glance.

Please remove the module state macros. They make this really difficult to review, and don't really add much (is MS_MEMBER(whatever) really that much easier to type than state->whatever?)

Modules/_zstd/zdict.c

Modules/_zstd/_zstdmodule.h

Modules/_zstd/compressor.c

Modules/_zstd/_zstdmodule.c

Modules/_zstd/compressor.c

emmatyping · 2025-04-26T19:58:34Z

Thank you for the review @ZeroIntensity! Some of the decisions are just how things were in pyzstd, so I expect they very well should change for merging the code in. I'll go through your review and read up on ob_lock.

ZeroIntensity · 2025-04-26T20:00:20Z

There's probably not too much to read about it other than somewhere in PEP-703. You'll just want to replace the locks with critical sections, which have some magic to prevent deadlocks.

Doc/license.rst

Modules/_zstd/_zstdmodule.h

Modules/_zstd/zdict.c

picnixz

Few preliminary comments, although I'm not sure it can be modified under the given license.

Modules/_zstd/_zstdmodule.c

picnixz

Some other comments I managed to find. I didn't review the decompressor but I think most of my comments that I said for the compressor would apply. The most important ones are essentially function signatures where I want you to use PyObject *self and do a manual cast to the expected type inside the function:

#define MyType_CAST(op)	((MyType *)(op))

static void
my_method(PyObject *op) {
    MyType *self = MyType_CAST(op);
    ...
}

The use of a macro (or a static inline function, up to you) is to allow future runtime type checks.

Modules/_zstd/_zstdmodule.c

Modules/_zstd/buffer.h

Modules/_zstd/compressor.c

Modules/_zstd/zdict.c

Doc/license.rst

Modules/_zstd/_zstdmodule.c

This commit introduces the `_zstd` module, with bindings to libzstd from the pyzstd project. It also includes the unix build system configuration. Windows build system support will be integrated independently as it depends on integration with cpython-source-deps.

Also removes module state references from the classes in the _zstd module and instead uses PyType_GetModuleState()

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

This should avoid races and deadlocks.

The `compress`/`decompress` functions will be moved to Python code for simplicity. C implementations can always be re-added in the future. Also, mark _zstd as not requiring the GIL.

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

bedevere-bot · 2025-05-03T16:26:05Z

🤖 New build scheduled with the buildbot fleet by @emmatyping for commit 30b3934 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133027%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

emmatyping · 2025-05-03T16:28:15Z

Okay, latest commit should actually work correctly on RHEL 8, I tested it in a container and verified _zstd gets disabled.

bedevere-bot · 2025-05-03T17:54:11Z

🤖 New build scheduled with the buildbot fleet by @emmatyping for commit 10e2e80 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133027%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

emmatyping · 2025-05-03T18:04:44Z

Okay, configure should now work across all platforms, that was a bit more involved than I was expecting. I wish the configure feedback loop wasn't so long :(

emmatyping · 2025-05-03T19:02:30Z

The RHEL8 s390x LTO+PGO failure is unrelated, should be #133261 (comment) (other RHEL 8 s390x may fail too for the same reason?)

But the changes have worked on RHEL8: https://buildbot.python.org/#/builders/420/builds/1735

checking for stdlib extension module _zstd... disabled

hugovk · 2025-05-03T19:34:36Z

Okay, configure should now work across all platforms, that was a bit more involved than I was expecting. I wish the configure feedback loop wasn't so long :(

I normally pass --config-cache to configure which makes it a lot quicker, 3s cached vs. 39s without.

emmatyping · 2025-05-03T19:38:50Z

I normally pass --config-cache to configure which makes it a lot quicker, 3s cached vs. 39s without.

TIL! Thanks that is really handy.

ZeroIntensity · 2025-05-04T01:38:18Z

Yay! Congrats for getting this accepted and into 3.14 on time @emmatyping

emmatyping · 2025-05-04T02:20:04Z

Thank you, as well as everyone who reviewed! The feedback was incredibly valuable. Now on to merging the Python code: #133365
:)

* origin/main: (111 commits) pythongh-91048: Add filename and line number to external inspection routines (pythonGH-133385) pythongh-131178: Add tests for `ast` command-line interface (python#133329) Regenerate pcbuild.sln in Visual Studio 2022 (python#133394) pythongh-133042: disable HACL* HMAC on Emscripten (python#133064) pythongh-133351: Fix remote PDB's multi-line block tab completion (python#133387) pythongh-109700: Improve stress tests for interpreter creation (pythonGH-109946) pythongh-81793: Skip tests for os.link() to symlink on Android (pythonGH-133388) pythongh-126835: Rename `ast_opt.c` to `ast_preprocess.c` and related stuff after moving const folding to the peephole optimizier (python#131830) pythongh-91048: Relax test_async_global_awaited_by to fix flakyness (python#133368) pythongh-132457: make staticmethod and classmethod generic (python#132460) pythongh-132805: annotationlib: Fix handling of non-constant values in FORWARDREF (python#132812) pythongh-132426: Add get_annotate_from_class_namespace replacing get_annotate_function (python#132490) pythongh-81793: Always call linkat() from os.link(), if available (pythonGH-132517) pythongh-122559: Synchronize C and Python implementation of the io module about pickling (pythonGH-122628) pythongh-69605: Add PyREPL import autocomplete feature to 'What's New' (python#133358) bpo-44172: Keep reference to original window in curses subwindow objects (pythonGH-26226) pythonGH-133231: Changes to executor management to support proposed `sys._jit` module (pythonGH-133287) pythongh-133363: Fix Cmd completion for lines beginning with `! ` (python#133364) pythongh-132983: Introduce `_zstd` bindings module (pythonGH-133027) pythonGH-91048: Add utils for printing the call stack for asyncio tasks (python#133284) ...

ned-deily

@emmatyping, @gpshead, @hugovk Some showstopper problems with trying to build _zstd.

ned-deily · 2025-05-05T20:24:49Z

configure.ac

@@ -5415,6 +5415,40 @@ PKG_CHECK_MODULES([LIBLZMA], [liblzma], [have_liblzma=yes], [
  ])
 ])

+have_libzstd=no
+AC_DEFUN([TEST_ZSTD_VERSION],[
+  AC_MSG_CHECKING([if libzstd is new enough])


There seem to be two problems with the new enough check:

There should be AC_MSG_RESULT([yes]) and AC_MSG_RESULT([no]) calls to complete the messages:

checking for libzstd... yes checking if libzstd is new enough... checking for hstrerror... yes

ned-deily · 2025-05-05T20:41:58Z

configure.ac

+  have_libzstd=yes
+])
+
+PKG_CHECK_MODULES([LIBZSTD], [libzstd], [TEST_ZSTD_VERSION()], [


and 2. There's something funky going on here with the two calls to TEST_ZSTD_VERSION() such that version number test fails and the failure is sort of hidden in config.log:

configure:22400: checking for libzstd configure:22407: $PKG_CONFIG --exists --print-errors "libzstd" configure:22410: $? = 0 configure:22424: $PKG_CONFIG --exists --print-errors "libzstd" configure:22427: $? = 0 configure:22727: result: yes configure:22730: checking if libzstd is new enough configure:22756: gcc -o conftest conftest.c -ldl >&5 conftest.c:325:10: fatal error: 'zstd.h' file not found 325 | #include "zstd.h" | ^~~~~~~~ 1 error generated. configure:22756: $? = 1 configure: program exited with status 1

This later results in:
checking for stdlib extension module _zstd... disabled

I suspect the problem is the call to TEST_ZSTD_VERSION() on line 5437 fails because at that point CPPFLAGS is no longer set to include $LIBZSTD_CFLAGS and the test compile would succeed only if the header files are installed in a default system location, like /usr/include, which won't be the case, for example, on builds on macOS.

Postscript: a likely confirmation of my suspicion above. For our CI runs on macOS, we use Homebrew to supply third-party libs (like libzstd) not supplied by Apple. If you look at CI logs for current PRs (i.e. after this PR was merged), _zstd is disabled (as above) on the ghcr.io/cirruslabs/macos-runner:sonoma runner but succeeds on the macos-13 runner. The difference is that the former is Apple Silicon (arm64) hardware where Homebrew installs to a non-standard /opt/homebrew prefix, whereas the macos-13 runner is Intel hardware where Homebrew continues to install to the legacy /usr/local prefix which, no doubt, is included in the compiler's default search path.

emmatyping · 2025-05-06T04:16:05Z

To follow up on autoconf issues Ned raised, those were addressed in #133479

emmatyping added the skip news label Apr 26, 2025

emmatyping requested a review from gpshead April 26, 2025 19:38

emmatyping requested review from erlend-aasland and corona10 as code owners April 26, 2025 19:38

bedevere-app bot added the awaiting review label Apr 26, 2025

bedevere-app bot mentioned this pull request Apr 26, 2025

Implement PEP 784 - Adding Zstandard to the Python standard library #132983

Open

11 tasks

emmatyping changed the title ~~gh-132983: Introduce _zstd module~~ gh-132983: Introduce _zstd bindings module Apr 26, 2025

ZeroIntensity reviewed Apr 26, 2025

View reviewed changes

emmatyping requested a review from ericsnowcurrently as a code owner April 26, 2025 21:01

StanFromIreland reviewed Apr 27, 2025

View reviewed changes

picnixz reviewed Apr 27, 2025

View reviewed changes

AA-Turner reviewed Apr 27, 2025

View reviewed changes

Doc/license.rst Outdated Show resolved Hide resolved

AA-Turner reviewed Apr 27, 2025

View reviewed changes

Modules/_zstd/_zstdmodule.c Outdated Show resolved Hide resolved

AA-Turner reviewed Apr 27, 2025

View reviewed changes

Modules/_zstd/_zstdmodule.c Outdated Show resolved Hide resolved

picnixz mentioned this pull request Apr 27, 2025

Show value of UINT32_MAX in HMAC error messages #133061

Closed

emmatyping and others added 11 commits April 28, 2025 17:52

Add _zstd module

9814e3b

This commit introduces the `_zstd` module, with bindings to libzstd from the pyzstd project. It also includes the unix build system configuration. Windows build system support will be integrated independently as it depends on integration with cpython-source-deps.

Add _zstd to modules

fda87c8

Fix path for compression.zstd module

887e564

Ignore _zstd module like _io

cdba656

Expand module state macros to improve code quality

6b67e9b

Also removes module state references from the classes in the _zstd module and instead uses PyType_GetModuleState()

Remove backticks suggested in review

a99a5d2

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

Use critical sections to lock object state

02cd17a

This should avoid races and deadlocks.

Remove compress/decompress and mark module as not reliant on the GIL

54eca74

The `compress`/`decompress` functions will be moved to Python code for simplicity. C implementations can always be re-added in the future. Also, mark _zstd as not requiring the GIL.

Lift critical section to avoid clang warning

f605956

Respond to comments by picnixz

2eadc65

Call out pyzstd explicitly in license description

8eac354

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

emmatyping added 2 commits May 3, 2025 08:35

Use AC_RUN_IFELSE

4a0a33e

Use a define() to re-use version check

30b3934

emmatyping added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 3, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 3, 2025

emmatyping added 3 commits May 3, 2025 09:41

Merge branch 'main' into 3.14-zstd-c-code

90b9172

Actually properly set _zstd module status based on version

8fdc7f7

Merge branch 'main' into 3.14-zstd-c-code

10e2e80

emmatyping added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 3, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 3, 2025

This comment was marked as abuse.

Sign in to view

hugovk mentioned this pull request May 3, 2025

Recommend --config-cache for configure python/devguide#1547

Open

Merge branch 'main' into 3.14-zstd-c-code

d90b29d

gpshead approved these changes May 4, 2025

View reviewed changes

gpshead enabled auto-merge (squash) May 4, 2025 01:24

gpshead merged commit 3b43335 into python:main May 4, 2025
42 checks passed

bedevere-app bot removed the awaiting merge label May 4, 2025

emmatyping deleted the 3.14-zstd-c-code branch May 4, 2025 01:36

ned-deily reviewed May 5, 2025

View reviewed changes

AA-Turner mentioned this pull request May 6, 2025

GH-132983: Remove zstd version check in the header file #133502

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-132983: Introduce `_zstd` bindings module #133027

gh-132983: Introduce `_zstd` bindings module #133027

emmatyping commented Apr 26, 2025 •

edited

Loading

emmatyping commented Apr 26, 2025 •

edited

Loading

ZeroIntensity left a comment

emmatyping commented Apr 26, 2025

ZeroIntensity commented Apr 26, 2025

picnixz left a comment

picnixz left a comment

bedevere-bot commented May 3, 2025

emmatyping commented May 3, 2025

bedevere-bot commented May 3, 2025

emmatyping commented May 3, 2025

This comment was marked as abuse.

emmatyping commented May 3, 2025

hugovk commented May 3, 2025

emmatyping commented May 3, 2025

ZeroIntensity commented May 4, 2025

emmatyping commented May 4, 2025

ned-deily left a comment •

edited

Loading

ned-deily May 5, 2025

ned-deily May 5, 2025 •

edited

Loading

ned-deily May 5, 2025

emmatyping commented May 6, 2025

gh-132983: Introduce _zstd bindings module #133027

gh-132983: Introduce _zstd bindings module #133027

Conversation

emmatyping commented Apr 26, 2025 • edited Loading

emmatyping commented Apr 26, 2025 • edited Loading

ZeroIntensity left a comment

Choose a reason for hiding this comment

emmatyping commented Apr 26, 2025

ZeroIntensity commented Apr 26, 2025

picnixz left a comment

Choose a reason for hiding this comment

picnixz left a comment

Choose a reason for hiding this comment

bedevere-bot commented May 3, 2025

emmatyping commented May 3, 2025

bedevere-bot commented May 3, 2025

emmatyping commented May 3, 2025

This comment was marked as abuse.

emmatyping commented May 3, 2025

hugovk commented May 3, 2025

emmatyping commented May 3, 2025

ZeroIntensity commented May 4, 2025

emmatyping commented May 4, 2025

ned-deily left a comment • edited Loading

Choose a reason for hiding this comment

ned-deily May 5, 2025

Choose a reason for hiding this comment

ned-deily May 5, 2025 • edited Loading

Choose a reason for hiding this comment

ned-deily May 5, 2025

Choose a reason for hiding this comment

emmatyping commented May 6, 2025

gh-132983: Introduce `_zstd` bindings module #133027

gh-132983: Introduce `_zstd` bindings module #133027

emmatyping commented Apr 26, 2025 •

edited

Loading

emmatyping commented Apr 26, 2025 •

edited

Loading

ned-deily left a comment •

edited

Loading

ned-deily May 5, 2025 •

edited

Loading