Python 3.8 / 3.9 dependency problems (werkzeug / gevent/greenlet)

[ssairanen]

Describe the bug

Dependencies that get downloaded and used by pywb have deprecated / changed something and current requirements.txt doesn't work out of the box in py 3.8, 3.9 (ubuntu)

Steps to reproduce the bug

Install pywb with python 3.8 / 3.9 ->

wayback -a
Traceback (most recent call last):
File "/usr/local/bin/wayback", line 11, in
load_entry_point('pywb==2.7.0b1', 'console_scripts', 'wayback')()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 490, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2854, in load_entry_point
return ep.load()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2445, in load
return self.resolve()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2451, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/usr/local/lib/python3.8/dist-packages/pywb-2.7.0b1-py3.8.egg/pywb/apps/cli.py", line 1, in
from gevent.monkey import patch_all; patch_all()
File "/usr/local/lib/python3.8/dist-packages/gevent-20.9.0-py3.8-linux-x86_64.egg/gevent/init.py", line 86, in
from gevent._hub_local import get_hub
File "/usr/local/lib/python3.8/dist-packages/gevent-20.9.0-py3.8-linux-x86_64.egg/gevent/_hub_local.py", line 101, in
import_c_accel(globals(), 'gevent.__hub_local')
File "/usr/local/lib/python3.8/dist-packages/gevent-20.9.0-py3.8-linux-x86_64.egg/gevent/_util.py", line 148, in import_c_accel
mod = importlib.import_module(cname)
File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "src/gevent/_hub_local.py", line 1, in init gevent._gevent_c_hub_local
ValueError: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 40 from PyObject

wayback --record --live -a --auto-interval 10

File "/usr/local/lib/python3.9/dist-packages/pywb-2.7.0b1-py3.9.egg/pywb/rewrite/default_rewriter.py", line 23, in
from werkzeug.useragents import UserAgent
ModuleNotFoundError: No module named 'werkzeug.useragents'

Expected behavior

pywb would run

Environment

Ubuntu / Rocky linux 8.5

Additional context

Installed greenlet isn't compatible with gevent
can be fixed with pip install greenlet==1.1.2

Installed werkzeug is version 2.1.0: New in version 2.0: This replaces the previous useragents module, but does not provide a built-in parser.
Can be fixed with pip install werkzeug==1.0.1

Should gevent be updated?
Should pywb code using werkzeug be fixed?

[edsu]

I'm testing the 2.7.0-beta branch and it looks like maybe this change hasn't been merged in yet? I'm hitting this error when uwsgi starts up:

was-pywb-pywb-1  | *** Operational MODE: preforking+async ***
was-pywb-pywb-1  | Traceback (most recent call last):
was-pywb-pywb-1  |   File "/usr/local/lib/python3.8/dist-packages/pywb/apps/wayback.py", line 2, in <module>
was-pywb-pywb-1  |     from pywb.apps.frontendapp import FrontEndApp
was-pywb-pywb-1  |   File "/usr/local/lib/python3.8/dist-packages/pywb/apps/frontendapp.py", line 26, in <module>
was-pywb-pywb-1  |     from pywb.apps.rewriterapp import RewriterApp
was-pywb-pywb-1  |   File "/usr/local/lib/python3.8/dist-packages/pywb/apps/rewriterapp.py", line 12, in <module>
was-pywb-pywb-1  |     from pywb.rewrite.default_rewriter import DefaultRewriter, RewriterWithJSProxy
was-pywb-pywb-1  |   File "/usr/local/lib/python3.8/dist-packages/pywb/rewrite/default_rewriter.py", line 23, in <module>
was-pywb-pywb-1  |     from werkzeug.useragents import UserAgent
was-pywb-pywb-1  | ModuleNotFoundError: No module named 'werkzeug.useragents'
was-pywb-pywb-1  | unable to load app 0 (mountpoint='') (callable not found or import error)
was-pywb-pywb-1  | *** no app loaded. going in full dynamic mode ***