MarkupSafe and Jinja pins are preventing building documentation for new versions of MarkupSafe, Jinja, and Flask

[davidism]

I am currently releasing MarkupSafe 2.0 and Jinja2 3.0. Sphinx depends on MarkupSafe<2.0 and Jinja2<3.0. We require Sphinx to build our docs, so we can't build the docs for MarkupSafe or Jinja2 until Sphinx stops pinning its dependency.

• https://markupsafe.palletsprojects.com/en/master/changes/#version-2-0-0
• https://jinja.palletsprojects.com/en/master/changes/#version-3-0-0

Running Sphinx v4.0.1
loading translations [en]... done
making output directory... done

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/sphinx/cmd/build.py", line 279, in build_main
    args.tags, args.verbosity, args.jobs, args.keep_going)
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/sphinx/application.py", line 276, in __init__
    self._init_builder()
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/sphinx/application.py", line 334, in _init_builder
    self.builder.init()
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/sphinx/builders/html/__init__.py", line 220, in init
    self.init_highlighter()
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/sphinx/builders/html/__init__.py", line 280, in init_highlighter
    self.highlighter = PygmentsBridge('html', style)
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/sphinx/highlighting.py", line 82, in __init__
    style = self.get_style(stylename)
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/sphinx/highlighting.py", line 99, in get_style
    return get_style_by_name(stylename)
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/pygments/styles/__init__.py", line 65, in get_style_by_name
    for found_name, style in find_plugin_styles():
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/pygments/plugin.py", line 64, in find_plugin_styles
    yield entrypoint.name, entrypoint.load()
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2449, in load
    self.require(*args, **kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2472, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/pkg_resources/__init__.py", line 777, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (MarkupSafe 2.0.0 (/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages), Requirement.parse('MarkupSafe<2.0'), {'Sphinx'})

Exception occurred:
  File "/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages/pkg_resources/__init__.py", line 777, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (MarkupSafe 2.0.0 (/home/docs/checkouts/readthedocs.org/user_builds/jinja/envs/1417/lib/python3.7/site-packages), Requirement.parse('MarkupSafe<2.0'), {'Sphinx'})
The full traceback has been saved in /tmp/sphinx-err-r8t2byv_.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!

[davidism]

This is also preventing building Flask 2.0's docs, since it depends on Jinja2>3.0.

[tk0miya]

@ericholscher Do you have any opinion on this unpinning? Would you like to test it before unpinning?

[ThiefMaster]

btw, #9162 already fixed it but only in the 4.x branch

[davidism]

We worked around this at the moment by pinning our Sphinx dependency to the 4.1 dev branch, but I would like to move back to a stable version when possible.

[ViktorHaag]

btw, #9162 already fixed it but only in the 4.x branch

Is it reasonable to request that this PR also get applied to the 4.0.x branch? My ad-hoc testing seems to demonstrate that the patch is also safe against the 4.0.x branch and doing so might get the updated dependencies into the stable build faster?

[ericholscher]

This seems like a larger issue that we need to understand. Sphinx itself depends on a set of versions that are needed to run the software. How would it work if the new version of MarkUpSafe broke the version of Sphinx needed? Is this just something that is impossible to handle because Sphinx requires importing code in order to generate docs for it?

[davidism]

While it's not possible to guarantee that a new release won't affect anything, I've tried to be much better about issuing deprecation warnings before removing things in a subsequent release. So it should have been safe to pin to Jinja2 < 2.1 instead of 2.0. I also plan to make release candidates before final feature releases going forward, and advertising that better, to allow more time for testing.

Additionally, we've started talking about how we might work with the wider ecosystem to run tests against release candidates to make sure a new release doesn't disrupt essential projects like Sphinx.

[ericholscher]

Additionally, we've started talking about how we might work with the wider ecosystem to run tests against release candidates to make sure a new release doesn't disrupt essential projects like Sphinx.

Indeed -- we're trying to work on this as well in the Sphinx ecosystem, which is how we ended up pinning our dependencies.

While it's not possible to guarantee that a new release won't affect anything, I've tried to be much better about issuing deprecation warnings before removing things in a subsequent release. So it should have been safe to pin to Jinja2 < 2.1 instead of 2.0. I also plan to make release candidates before final feature releases going forward, and advertising that better, to allow more time for testing.

Gotcha -- I understand that hopefully the release won't break things, but I think pinning to the latest released version is a good practice for both security and stability. Arguably we should be moving towards a place where we are actually pinning a hash for the packaged version or similar, and validating that when upgrading dependencies. That's a longer term ideal, and not too relevant currently.

I feel like the real bug issue here is figuring out how to generate docs for a Python library version different than one that is installed for Sphinx's use. I feel like I haven't hit this issue at all in the past, so I'm not quite sure why it would be happening now. Is Python enforcing the installed library requirements on import of the package, and that is what is breaking here? That seems like a weird outcome, and something that would effect the larger ecosystem, but we haven't seen reports of it.

[davidism]

It looks like you're using pkg_resources to load some things, probably extensions with version requirements. I'm not sure, but perhaps importlib_metadata does not have the same requirement that the versions actually match? We've transitioned from pkg_resources to importlib_metadata, it would probably be good to explore that in Sphinx even if it doesn't end addressing the version issue.

[ericholscher]

@tk0miya FWIW, I recommend that we bump to the latest released versions, but I think this is still a symptom of a larger issue that we need to figure out, or at least document.

[StevenMapes]

This is now worse since Jinja3 version 3.0.x is out I just hit dependency hell which killed pip so I've had to remove sphinx from my app now because I need the newer versions of Jinja and MarkupSafe

[tk0miya]

@ericholscher Thank you for your advice. I just posted #9251.

[davidism]

Thanks, just rebuilt MarkupSafe with Sphinx 4.0.2 and Jinja 3.0.1, everything's working.