Use the latest Sphinx version

[AA-Turner]

Documentation

Python's documentation uses the Sphinx toolchain. Originally written for the Python documentation, it is now developed and distributed as an independent project. The two remain closely linked, e.g. when new syntax is added to the Python language. To document this syntax in the best way, Sphinx must support it and Python must require a version of Sphinx that supports the syntax.

Using the new syntax in PEP-695 as an example:

1. The implementation was committed on 16 May (gh-103763: Implement PEP 695 #103764)
2. The first draft of the documentation was written on 19 May (gh-103921: Document PEP 695 #104642)
3. A feature request was created in Sphinx on 23 May (Support PEP 695 syntax for class definitions sphinx-doc/sphinx#11438)
4. Sphinx added support on 23 July (Support PEP 695 and PEP 696 syntax in py:function and py:class directives. sphinx-doc/sphinx#11444)
5. Sphinx published a release with support on 24 July (v7.1.0)
6. The Python project can use those features, 19 October [+ 1 year] (GH-125277: Increase minimum supported Sphinx to 7.2.6 #125368)

This is a ~two month window from implementation to support in a release of Sphinx. It took a further 15 months and two feature releases (3.12, 3.13) until these features can be used in Python. Due to this, our documentation is meaningfully worse for readers and programmers. Using older versions of Sphinx mean that we cannot succinctly cover the features and syntax that exist within released versions of Python.

Core developers responsible for these features have expressed interest, but have been hampered by our self-imposed restriction of the minimum version of Sphinx that we support.

We adopt these restrictions for the benefit of downstream Linux redistributors, as can be seen in the more recent issues in the summary table below. This has not always been the case. From the adoption of Sphinx in 2007 (8ec7f65 / 116aa62) until 2014 (f7b2f36), the latest source-tree checkout of Sphinx at https://svn.python.org/ was used. From then until ~2018 the minimum required Sphinx version (controlled by needs_sphinx in conf.py) tracked the latest release promptly. This has since ossified.

In a recent informal discussion, several committers supported the idea of removing or relaxing this restriction, allowing the Python documentation to use the latest version(s) of Sphinx. This is the context for this note, which for now is a proposal. The status quo will prevail if there is not sufficient support.

As a concrete suggestion, I propose that when evaulating increasing the minimum required version of Sphinx, we no longer consult downstream redistributors' Sphinx versions. The procedure will follow standard Python development processes in all other ways. I expect that the minimum version would be updated if and when a new Sphinx feature is of sufficient benefit to Python. This, though, may be a greater version than a downstream redistributor provides.

We would like to solicit views from representatives of downstream redistributors as to how (in)feasible this proposal would be. My understanding is that Fedora and SUSE have processes whereby a newer version of Sphinx can be used soley for the Python documentation. I do not know how this will impact Debian or Gentoo.

Thank you in advance for your consideration,
Adam

Table of past changes to needs_sphinx

needs_sphinx	Issue	PR / Commit	Latest Sphinx 1
7.2.6	#125277	#125368	8.1.3
6.2.1	#117928	#121986	7.4.6
4.2	#109209	#109210	7.2.6
3.2	#86986	#93337	5.0.1
1.8	#80188	#11887	1.8.4
1.7	doc-sig	#9423	1.8.1
1.2	#65630	90d76ca	1.2.3
1.1	#64860	f7b2f36	1.2.1

Previous discussion relating to the minumum version:

• Docs build error with Sphinx 3.0 due to invalid C declaration #84385
• What min_sphinx for Python 3.10 #87009
• Require Sphinx 6.2 to build Python 3.13 documentation #104818

cc:

• @doko42 @mitya57 / Debian
• @hroncok / Fedora / RHEL
• @mgorny / Gentoo
• @danigm @mcepl / openSUSE
• @AA-Turner @hugovk / CPython

Footnotes

1. At commit date. ↩

[willingc]

@AA-Turner It would be helpful to have the latest version of Sphinx build the latest release of CPython. I suspect most users will view documentation on python.org.

[AA-Turner]

Yep, and we currently use the latest version of Sphinx on docs.python.org (see the footer for "Created using ..."). This proposal would in effect mean we can narrow the range between the minimum supported and what we use in Doc/requirements.txt.

A

[hroncok]

From the Fedora point of view, we have no trouble allowing the latest/development Python 3.X version documentation to use the latest version of Sphinx. My understanding it that this is usually the only Python version that needs to use the new features (such as new Python syntax). Once the features of a Python branch are stable, the required Sphinx version IMHO should also remain stable.

In the past, you proposed bumping the minimum Sphinx version for older Python versions, such as 3.12. That might sometimes be problematic for us. That said, your communication wrt. such bumps was always appreciated but never taken for granted. If you decide to always use the latest Sphinx for Python 3.12, 3.12, etc. at this point, it might be a bit challenging for us, but it won't be a blocker.

[AA-Turner]

Thank you @hroncok, this is good to know.

In the past, you proposed bumping the minimum Sphinx version for older Python versions, such as 3.12. That might sometimes be problematic for us. [...] If you decide to always use the latest Sphinx for Python 3.12, 3.12, etc. at this point, it might be a bit challenging for us, but it won't be a blocker.

Our development workflow means that we backport almost all documentation PRs to stable versions of Python (currently 3.12 & 3.13). This is mainly so that the website at https://docs.python.org/3/ shows the latest documentation, as /3/ is a symlink to the most recent stable (currently /3.13).

This would mean that we would increase the minimum version for both HEAD and also all bugfix branches (or at least the most recent bugfix branch). The proposal wouldn't have much value without this, as it would effectively prohibit us from being able to confidently backport PRs.

Hopefully this is a little clearer. I think that it falls into the 'challenging but manageable' category of your response, if I'm understanding correctly.

A