werkzeug 2.1.0 removal of useragents module causes error starting wayback

[tw4l]

Describe the bug

The 2.1.0 release of werkzeug removed the werkzeug.useragents module, which pywb uses in the rewriterapp. This causes the wayback command to throw a 500 Internal Server error.

Stacktrace:

pwyb_1  | Traceback (most recent call last):
pwyb_1  |   File "/usr/local/lib/python3.7/site-packages/pywb-2.7.0-py3.7.egg/pywb/apps/wayback.py", line 2, in <module>
pwyb_1  |     from pywb.apps.frontendapp import FrontEndApp
pwyb_1  |   File "/usr/local/lib/python3.7/site-packages/pywb-2.7.0-py3.7.egg/pywb/apps/frontendapp.py", line 26, in <module>
pwyb_1  |     from pywb.apps.rewriterapp import RewriterApp
pwyb_1  |   File "/usr/local/lib/python3.7/site-packages/pywb-2.7.0-py3.7.egg/pywb/apps/rewriterapp.py", line 12, in <module>
pwyb_1  |     from pywb.rewrite.default_rewriter import DefaultRewriter, RewriterWithJSProxy
pwyb_1  |   File "/usr/local/lib/python3.7/site-packages/pywb-2.7.0-py3.7.egg/pywb/rewrite/default_rewriter.py", line 23, in <module>
pwyb_1  |     from werkzeug.useragents import UserAgent
pwyb_1  | ModuleNotFoundError: No module named 'werkzeug.useragents'

Steps to reproduce the bug

• Build the 2.7.0-beta branch of pywb (likely other branches are affected as well), and run wayback, e.g. with docker-compose
• Go to localhost:8080 in the browser
• See 500 Internal Server Error and inspect logs for traceback

Expected behavior

Screenshots

Environment

• OS: [e.g. iOS]: n/a
• Browser [e.g. chrome, safari]: n/a
• Version [e.g. 22]: Noticed in branch 2.7.0-beta

Additional context

The issue can be temporarily avoided by pinning werkzeug to werkzeug==2.0.3 in the requirements file. For a more proper fix it looks like pywb should be modified to use the werkzeug.user_agent module instead.

[ikreymer]

This was not merged into the 2.7.0 branch, but actually removed werkzeug.user_agent on mainline in this PR:
#708

We can merge this back into 2.7.0 after the next point release.

[tw4l]

Closing, as rebasing the 2.7.0 dev branch on the 2.6.8 release made this issue obsolete.