An os.sched_param object can be dumped with pickle, but loading raises TypeError

[Willy19921992]

Bug report

Bug description:

While working on an experiment with different scheduler priorities and the multiprocessing.Pool, I found out that using an os.sched_param object as one of the initargs parameters raised the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/forkserver.py", line 274, in main
    code = _serve_one(child_r, fds,
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/forkserver.py", line 313, in _serve_one
    code = spawn._main(child_r, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: sched_param() takes at most 1 argument (2 given)

With a bit of investigation using the REPL, I quickly found out a simple way to reproduce the problem:

Python 3.12.7 (main, Oct  1 2024, 11:15:50) [GCC 14.2.1 20240910] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import pickle
>>> with open('schedparam', 'wb') as f:
...     pickle.dump(os.sched_param(99), f)
... 
>>> with open('schedparam', 'rb') as f:
...     pickle.load(f)
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: sched_param() takes at most 1 argument (2 given)

The exact value used in the os.sched_param constructor doesn't affect the result. My specific use case is in Linux, where I'm trying to create a pool of processes where each one uses os.sched_setscheduler as an initializer to change their own scheduling policy to something different from the parent process and see the resulting behavior.

Im using Arch Linux with the package python3 3.12.7-1, which is the latest version as the time of writting. I have also found that this issue is also present in Pypy, if it is of any use.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-126303: Fix pickling and copying of os.sched_param objects #126336
• [3.13] gh-126303: Fix pickling and copying of os.sched_param objects (GH-126336) #126423
• [3.12] gh-126303: Fix pickling and copying of os.sched_param objects (GH-126336) #126424

[xooxo]

I took a look into this with pickletools (on Debian 12 Python 3.11.2). It seems that the problem is pickle creates tuple with 99 and an empty dictionary for keyword arguments.

>>> import pickletools
>>> with open("schedparam", "rb") as f:
...     pickletools.dis(f)
... 
    0: \x80 PROTO      4
    2: \x95 FRAME      35
   11: \x8c SHORT_BINUNICODE 'posix'
   18: \x94 MEMOIZE    (as 0)
   19: \x8c SHORT_BINUNICODE 'sched_param'
   32: \x94 MEMOIZE    (as 1)
   33: \x93 STACK_GLOBAL
   34: \x94 MEMOIZE    (as 2)
   35: K    BININT1    99
   37: \x85 TUPLE1
   38: \x94 MEMOIZE    (as 3)
   39: }    EMPTY_DICT
   40: \x94 MEMOIZE    (as 4)
   41: \x86 TUPLE2
   42: \x94 MEMOIZE    (as 5)
   43: R    REDUCE
   44: \x94 MEMOIZE    (as 6)
   45: .    STOP

[picnixz]

cc @serhiy-storchaka as the pickle expert

[serhiy-storchaka]

os.sched_param is a type created by PyStructSequence_NewType(). But it has special constructor, different from common PyStructSequence constructors. The __reduce__ method returns value that works with standard PyStructSequence constructor.

There are two ways to fix this issue:

• Change the custom constructor to accept also PyStructSequence-compatible signature.
• Override the __reduce__ method.

The former is more complex and error-prone. The latter needs adding internal C API for overriding methods specified by PyMethodDef. I hope this is enough.