Worker fails to run with Python 3.11

[saschamarkus-uberresearch]

When migrationg a batch pipeline with multiple workers to Python 3.11 the workers fails to run.

Stacktrace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/pack/app/venv/lib/python3.11/site-packages/luigi/worker.py", line 297, in run
    super(ContextManagedTaskProcess, self).run()
  File "/pack/app/venv/lib/python3.11/site-packages/luigi/worker.py", line 168, in run
    random.seed((processID, currentTime))
  File "/usr/local/lib/python3.11/random.py", line 160, in seed
    raise TypeError('The only supported seed types are: None,\n'
TypeError: The only supported seed types are:

The error is caused by a change in random.seed() for Python 3.11

Changed in version 3.11: The seed must be one of the following types: NoneType, int, float, str, bytes, or bytearray.

The issue can be reproduced by running
pytest -c /dev/null test/worker_task_process_test.py
which fails on test_context_manager_with_multiprocessing

One possible fix would be to replace
random.seed((processID, currentTime))
with something like
random.seed(processID * currentTime)
or
random.seed(str((processID, currentTime)))

[jmrnilsson]

Hmm, I hit this too. Let's check for new version..

[jmrnilsson]

No reverting back to Python 3.9 works though. So it's a work-around..

[dlstadther]

@jmrnilsson , the patch that @davidesba submitted in currently in master if you want to try it. It'll be included in a future (date tbd) versioned release of the project.

[saschamarkus-uberresearch]

@jmrnilsson
If you use the pyproject.toml/setup.cfg way to set up your project you can use the master branch from git instead of the release from pypy with this in your setup.cfg. There ist probably a similar way for the requirements.txt

[options]
packages = find:
install_requires =
  luigi @ git+https://github.com/spotify/luigi.git