Build Python 3.13 wheels (not free-threaded) #711

edgarrmondragon · 2024-07-13T06:55:27Z

I can close this if #703 gets more attention, but I've confirmed changes here allow the wheels to be built in 3.8-3.13, and all tests are passing.

This warning is currently raised, but fixing it might be beyond the scope of this PR:

PytestUnraisableExceptionWarning: Exception ignored in PyDict_GetItem(); consider using PyDict_GetItemRef() or PyDict_GetItemWithError(): None

python/cpython#106004

Closes 3.13 support #698
Closes Doesn't install on python 3.13 #741

edgarrmondragon

I think this is ready for an initial review.

Mostly followed advice from

.github/workflows/ci.yml

edgarrmondragon · 2024-07-13T07:04:21Z

msgspec/_utils.py

@@ -127,6 +127,7 @@ def get_class_annotations(obj):
                value = type(None)
            elif isinstance(value, str):
                value = _forward_ref(value)
+            # TODO: Pass `type_params` to `_eval_type` to fix deprecation warning


This missing parameter is causing a ton of warnings like this:

msgspec/_utils.py:130: DeprecationWarning: Failing to pass a value to the 'type_params' parameter of 'typing._eval_type' is deprecated, as it leads to incorrect behaviour when calling typing._eval_type on a stringified annotation that references a PEP 695 type parameter. It will be disallowed in Python 3.15.

python/cpython#118418

edgarrmondragon · 2024-07-13T19:45:15Z

This warning is currently raised but fixing it might be beyond the scope of this PR to fix it:
PytestUnraisableExceptionWarning: Exception ignored in PyDict_GetItem(); consider using PyDict_GetItemRef() or PyDict_GetItemWithError(): None

FWIW I think this is hiding an actual error:

PytestUnraisableExceptionWarning: Exception ignored in PyDict_GetItem(); consider using PyDict_GetItemRef() or PyDict_GetItemWithError(): None
    
    Traceback (most recent call last):
      File "/opt/_internal/cpython-3.13.0b3/lib/python3.13/typing.py", line 2251, in __hash__
        return hash((self.__origin__, self.__metadata__))
    TypeError: unhashable type: 'set'
    
      warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

in

msgspec/tests/test_inspect.py

Lines 790 to 793 in 2c37da0

    
           def test_inspect_with_unhashable_metadata(): 
        
               typ = Annotated[int, {"unhashable"}] 
        
               assert mi.type_info(typ) == mi.IntType()

.github/workflows/ci.yml

edgarrmondragon · 2024-07-21T17:39:51Z

macos-11 runners aren't available anymore, so I updated it to macos-14 in 748f4bb.

https://github.blog/changelog/2024-05-20-actions-upcoming-changes-to-github-hosted-macos-runners/#macos-11-deprecation-and-removal

edgarrmondragon · 2024-07-21T17:48:19Z

macos-11 runners aren't available anymore, so I updated it to macos-14 in 748f4bb.

https://github.blog/changelog/2024-05-20-actions-upcoming-changes-to-github-hosted-macos-runners/#macos-11-deprecation-and-removal

With that change, the wheels are finally built instead of the workflow getting stuck forever:

 12 wheels produced in 5 minutes:
  msgspec-0+untagged.1.g766a8f7-cp310-cp310-macosx_10_9_x86_64.whl    186 kB
  msgspec-0+untagged.1.g766a8f7-cp310-cp310-macosx_11_0_arm64.whl     180 kB
  msgspec-0+untagged.1.g766a8f7-cp311-cp311-macosx_10_9_x86_64.whl    184 kB
  msgspec-0+untagged.1.g766a8f7-cp311-cp311-macosx_11_0_arm64.whl     178 kB
  msgspec-0+untagged.1.g766a8f7-cp312-cp312-macosx_10_9_x86_64.whl    186 kB
  msgspec-0+untagged.1.g766a8f7-cp312-cp312-macosx_11_0_arm64.whl     180 kB
  msgspec-0+untagged.1.g766a8f7-cp313-cp313-macosx_10_13_x86_64.whl   186 kB
  msgspec-0+untagged.1.g766a8f7-cp313-cp313-macosx_11_0_arm64.whl     180 kB
  msgspec-0+untagged.1.g766a8f7-cp38-cp38-macosx_10_9_x86_64.whl      186 kB
  msgspec-0+untagged.1.g766a8f7-cp38-cp38-macosx_11_0_arm64.whl       180 kB
  msgspec-0+untagged.1.g766a8f7-cp39-cp39-macosx_10_9_x86_64.whl      186 kB
  msgspec-0+untagged.1.g766a8f7-cp39-cp39-macosx_11_0_arm64.whl       180 kB

https://github.com/edgarrmondragon/msgspec/actions/runs/10030128227/job/27719084757

tacaswell · 2024-07-25T16:26:03Z

I can also get msgspec to build with py313 with this branch, but not the branch in #703

edgarrmondragon · 2024-07-30T19:09:38Z

@jcrist This is ready for review at your convenience :)

edgarrmondragon · 2024-08-06T14:45:40Z

Updated to use Python 3.13.0rc1

edgarrmondragon · 2024-09-17T03:25:34Z

Updated to use Python 3.13.0rc2

ofek · 2024-09-18T12:56:37Z

@jcrist Can you please run the CI?

Tishka17 · 2024-10-02T18:23:28Z

Python 3.13 release is scheduled for then upcoming Monday. Are there any plans to merge this PR and publish a new version?

Tishka17 · 2024-10-07T16:55:30Z

Python 3.13 is out there.

maurerle · 2024-10-11T17:19:00Z

This closes #741 and #698

I was building on debian 13 trixie which uses GCC 14, I therefore had a few more required adjustments on top of yours:

maurerle@2aa5db5

To build this locally, I used conda:

conda create -n py313 python-freethreading -c conda-forge
conda activate pyt313
git clone -b py313 git@github.com:maurerle/msgspec.git
cd msgspec
pip install .

maurerle · 2024-10-11T17:42:33Z

I do get a segfault though when importing msgspec.. Even with GIL enabled

(py313) ➜  msgspec git:(py313) ✗ python -X gil=1 -X faulthandler
Python 3.13.0 experimental free-threading build | packaged by conda-forge | (main, Oct  8 2024, 20:16:19) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import msgspec
Fatal Python error: Segmentation fault

Current thread 0x00007fcd5e8ba640 (most recent call first):
  File "~/github/msgspec/msgspec/__init__.py", line 19 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<python-input-1>", line 1 in <module>

edgarrmondragon · 2024-10-12T00:31:30Z

I do get a segfault though when importing msgspec.. Even with GIL enabled

(py313) ➜  msgspec git:(py313) ✗ python -X gil=1 -X faulthandler
Python 3.13.0 experimental free-threading build | packaged by conda-forge | (main, Oct  8 2024, 20:16:19) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import msgspec
Fatal Python error: Segmentation fault

Current thread 0x00007fcd5e8ba640 (most recent call first):
  File "~/github/msgspec/msgspec/__init__.py", line 19 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1022 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<python-input-1>", line 1 in <module>

@maurerle thanks for finding that. I'll give a read to https://py-free-threading.github.io/porting/, unless someone beats me to it and knows what's wrong here.

maurerle · 2024-10-12T07:36:08Z

When executing this on a gil build (e.g. conda create -n py313-gil python=3.13)
One gets an error if executed from the build directory:

>>> import msgspec
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    import msgspec
  File "~/github/msgspec/msgspec/__init__.py", line 1, in <module>
    from ._core import (
    ...<14 lines>...
    )
ModuleNotFoundError: No module named 'msgspec._core'

So make sure to execute this not in the build directory :)
It works fine then :)
But I don't have a clue how to solve the above segfault either.

The segfault also occurs when starting the above mentioned freethreading build with an active GIL as in
python -X gil=1 -c "import msgspec"

Keeps around the semantic meaning for when this function is added back in Python 3.14.

jcrist · 2024-10-13T19:27:25Z

Thanks for the PR (and everyone for their patience). With a new kid and recent layoff, I've been a bit busy and msgspec maintenance has languished. I really appreciate the help and patience here.

I have a few small fixups I want to land with this release, but I'd expect a release supporting Python 3.13 (not with freethreading) out in the next day or two. I'll work on adding freethreading support after getting that initial release out.

ekneg54 · 2024-10-19T18:30:15Z

Hi,

with this merged I thought that I should be able to install msgspec on python 3.13. But it turns out, that this is not the case as you could see:

I wonder if I am missing something. Could you help?

maurerle · 2024-10-19T18:40:19Z

You are missing the python3-dev dependency.
Install it using your system package manager

mj0nez · 2024-10-19T18:42:54Z

It was merged but not released or am I mistaken? An install from main should work for sure: install from vcs

ekneg54 · 2024-10-19T18:51:15Z

thank you. I wait for the release. Do you know when it is planned?

mj0nez · 2024-10-19T19:04:56Z

No, but soon I guess.

edgarrmondragon force-pushed the py313 branch 3 times, most recently from dfca216 to 58a7826 Compare July 13, 2024 07:03

edgarrmondragon commented Jul 13, 2024

View reviewed changes

edgarrmondragon marked this pull request as ready for review July 13, 2024 07:08

edgarrmondragon commented Jul 13, 2024

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

edgarrmondragon mentioned this pull request Jul 15, 2024

chore: 3.13 support #703

Closed

edgarrmondragon mentioned this pull request Aug 6, 2024

3.13 support #698

Closed

This was referenced Aug 18, 2024

Install and run pytype on Windows and Python 3.12 python/typeshed#12547

Closed

Please update for Python 3.12. google/pytype#1475

Open

edgarrmondragon force-pushed the py313 branch from 1aff991 to 8f03317 Compare September 17, 2024 03:23

edgarrmondragon mentioned this pull request Oct 2, 2024

refactor: Implement msgspec encoding meltano/sdk#2541

Draft

3 tasks

reneleonhardt mentioned this pull request Oct 3, 2024

Add type hints for the get_unused_code function and the fields of Item. jendrikseipp/vulture#361

Merged

4 tasks

markmcd mentioned this pull request Oct 8, 2024

Add python 3.13 tests, fixes:#518 google-gemini/generative-ai-python#519

Open

hauntsaninja mentioned this pull request Oct 11, 2024

third-party test: Update Python versions python/typing_extensions#484

Merged

Build Python 3.13 wheels (not free-threaded)

765d332

edgarrmondragon and others added 14 commits October 13, 2024 13:02

Include cp313 wheels for testing

99be1eb

Build Python 3.13 wheels only for testing

53c24ea

Remove trove classifier

3c63ad9

Update for Python 3.13.0rc1

b01c5a8

Clean up

320a4d8

Bump cibuildwheel to 2.21.1

b06d771

Upgrade deprecated artifact actions

170e9f2

Update for Python 3.13.0rc3

9e0d3b5

Bump cibuildwheel to 2.21.3 to use Python 3.13.0 final

d35f64f

Squash _eval_type warning

67061e6

Still use _PyUnicode_EQ for older pythons

fd8b9dc

Keeps around the semantic meaning for when this function is added back in Python 3.14.

Use PyObject_GetIter for all versions

f20630c

Use PyLong_AsNativeBytes on Py3.13

a0cc7ea

Suppress hashing error when parsing annotated types

36e56c4

jcrist force-pushed the py313 branch from 4d71289 to 36e56c4 Compare October 13, 2024 19:17

jcrist merged commit 7ade469 into jcrist:main Oct 13, 2024
7 checks passed

jcrist mentioned this pull request Oct 13, 2024

Support for Python 3.13 freethreading #744

Open

5 tasks

edgarrmondragon deleted the py313 branch October 14, 2024 00:36

No767 mentioned this pull request Nov 5, 2024

Add Python 3.13 support No767/Catherine-Chan#220

Merged

9 tasks

macbre mentioned this pull request Nov 19, 2024

Add msgspec package pigs-will-fly/docker-python#114

Merged

bitwise-constructs mentioned this pull request Dec 2, 2024

feat: support python 3.13 ApeWorX/ape#2393

Merged

3 tasks

ncoghlan mentioned this pull request Dec 9, 2024

Migration to Python 3.13 #777

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build Python 3.13 wheels (not free-threaded) #711

Build Python 3.13 wheels (not free-threaded) #711

edgarrmondragon commented Jul 13, 2024 •

edited

Loading

edgarrmondragon left a comment

edgarrmondragon Jul 13, 2024 •

edited

Loading

edgarrmondragon commented Jul 13, 2024 •

edited

Loading

edgarrmondragon commented Jul 21, 2024

edgarrmondragon commented Jul 21, 2024

tacaswell commented Jul 25, 2024

edgarrmondragon commented Jul 30, 2024

edgarrmondragon commented Aug 6, 2024

edgarrmondragon commented Sep 17, 2024

ofek commented Sep 18, 2024

Tishka17 commented Oct 2, 2024 •

edited

Loading

Tishka17 commented Oct 7, 2024

maurerle commented Oct 11, 2024 •

edited

Loading

maurerle commented Oct 11, 2024 •

edited

Loading

edgarrmondragon commented Oct 12, 2024

maurerle commented Oct 12, 2024 •

edited

Loading

jcrist commented Oct 13, 2024

ekneg54 commented Oct 19, 2024

maurerle commented Oct 19, 2024

mj0nez commented Oct 19, 2024 •

edited

Loading

ekneg54 commented Oct 19, 2024

mj0nez commented Oct 19, 2024

Build Python 3.13 wheels (not free-threaded) #711

Build Python 3.13 wheels (not free-threaded) #711

Conversation

edgarrmondragon commented Jul 13, 2024 • edited Loading

edgarrmondragon left a comment

Choose a reason for hiding this comment

edgarrmondragon Jul 13, 2024 • edited Loading

Choose a reason for hiding this comment

edgarrmondragon commented Jul 13, 2024 • edited Loading

edgarrmondragon commented Jul 21, 2024

edgarrmondragon commented Jul 21, 2024

tacaswell commented Jul 25, 2024

edgarrmondragon commented Jul 30, 2024

edgarrmondragon commented Aug 6, 2024

edgarrmondragon commented Sep 17, 2024

ofek commented Sep 18, 2024

Tishka17 commented Oct 2, 2024 • edited Loading

Tishka17 commented Oct 7, 2024

maurerle commented Oct 11, 2024 • edited Loading

maurerle commented Oct 11, 2024 • edited Loading

edgarrmondragon commented Oct 12, 2024

maurerle commented Oct 12, 2024 • edited Loading

jcrist commented Oct 13, 2024

ekneg54 commented Oct 19, 2024

maurerle commented Oct 19, 2024

mj0nez commented Oct 19, 2024 • edited Loading

ekneg54 commented Oct 19, 2024

mj0nez commented Oct 19, 2024

edgarrmondragon commented Jul 13, 2024 •

edited

Loading

edgarrmondragon Jul 13, 2024 •

edited

Loading

edgarrmondragon commented Jul 13, 2024 •

edited

Loading

Tishka17 commented Oct 2, 2024 •

edited

Loading

maurerle commented Oct 11, 2024 •

edited

Loading

maurerle commented Oct 11, 2024 •

edited

Loading

maurerle commented Oct 12, 2024 •

edited

Loading

mj0nez commented Oct 19, 2024 •

edited

Loading