Skip to content

Python 3.11.5 fails with stix-shifter interface #401

@subbyte

Description

@subbyte

Describe the bug
Unit test hangs for anything that uses stix-shifter, e.g., test_cli.py or test_command_get.py::test_get_single_stixshifter_stix_bundle.

Details of the bug

Process TransmitterPool-3:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/Users/subx/venv/py311/lib/python3.11/site-packages/kestrel_datasource_stixshifter/worker/transmitter.py", line 53, in run
    transmitter.start()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/synchronize.py", line 106, in __getstate__
    if self.is_fork_ctx:
       ^^^^^^^^^^^^^^^^
AttributeError: 'Lock' object has no attribute 'is_fork_ctx'

To Reproduce
pytest

Expected behavior
No error.

Environment (please complete the following information):

  • Kestrel 1.7.5
  • macOS
  • Python 3.11.5 (latest as of 2023/09/12)

Solution
This looks like a bug in Python multiprocessing library: python/cpython#108520 and it is already solved and await to be released in the next Python version.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions