⬆️ Support Sphinx v9

[chrisjsewell]

This PR adds support for Sphinx version 9 while maintaining backward compatibility with Sphinx 8. The changes update dependency specifications, test configurations, and test fixtures to accommodate behavioral differences between the two Sphinx versions.

Changes:

• Updated Sphinx version constraints from <9 to <10 in dependencies
• Added Sphinx 9 to the test matrix and tox environments
• Updated test fixtures and test code to handle differences in Sphinx 8 vs 9 output

[chrisjsewell]

Currently, the major blocker appears to be that this function no longer seems to work properly:

MyST-Parser/myst_parser/mdit_to_docutils/base.py

Lines 1884 to 1929 in 59d5384

	def html_meta_to_nodes(
	data: dict[str, Any], document: nodes.document, line: int, reporter: Reporter
	) -> list[nodes.pending | nodes.system_message]:
	"""Replicate the `meta` directive,
	by converting a dictionary to a list of pending meta nodes
	See:
	https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#html-metadata
	"""
	if not data:
	return []
	output = []
	for key, value in data.items():
	content = str(value or "")
	meta_node = nodes.meta(content)
	meta_node.source = document["source"]
	meta_node.line = line
	meta_node["content"] = content
	try:
	if not content:
	raise ValueError("No content")
	for i, key_part in enumerate(key.split()):
	if "=" not in key_part and i == 0:
	meta_node["name"] = key_part
	continue
	if "=" not in key_part:
	raise ValueError(f"no '=' in {key_part}")
	attr_name, attr_val = key_part.split("=", 1)
	if not (attr_name and attr_val):
	raise ValueError(f"malformed {key_part}")
	meta_node[attr_name.lower()] = attr_val
	except ValueError as error:
	msg = reporter.error(f'Error parsing meta tag attribute "{key}": {error}.')
	output.append(msg)
	continue
	pending = nodes.pending(
	Filter,
	{"component": "writer", "format": "html", "nodes": [meta_node]},
	)
	document.note_pending(pending)
	output.append(pending)
	return output

It leads to:

      File ".../docutils/transforms/components.py", line 50, in apply
        component = self.document.transformer.components[component_type]
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
    KeyError: 'writer'

Not sure what has changed to affect this yet, feedback / help welcome?

[AA-Turner]

@chrisjsewell Fairly sure this is a Docutils change rather than Sphinx. Notably Docutils doesn't seem to use Filter anywhere and the docstring notes it as unused & marked for removal.

Docutils currently uses the below code to add meta nodes:

https://github.com/docutils/docutils/blob/5d129e11e46967418b884019fc3855529a5d74cf/docutils/docutils/parsers/rst/directives/misc.py#L632-L635

Could you do similar here?

A

[chrisjsewell]

Fairly sure this is a Docutils change rather than Sphinx

@AA-Turner the only change in dependencies is sphinx 8.2.3 to 9.0.4 (plus the addition of roman-numerals==4.0.0), the docutils version remains the same at 0.21.2 (docutils is stilled pinned <0.22)

is there anything you can think of that would change how docutils is initalised?

[chrisjsewell]

is there anything you can think of that would change how docutils is initalised?

@AA-Turner perhaps it is something to do with sphinx-doc/sphinx#13673

[alexlancaster]

I noticed in the CI logs above, that the combination of Python 3.14 and Sphinx 9.0.4 does seem to pass on Windows for some reason: https://github.com/executablebooks/MyST-Parser/actions/runs/20259242759/job/58167505565?pr=1076 does this mean there might be some Python version and Sphinx interaction?

[chrisjsewell]

does seem to pass on Windows for some reason

It's not actually passing, if you look in the log, it's an issue in the windows job step, that I've seen before; it doesn't "fail-fast" like Linux and takes the fact that the coverage xml command passes as the whole step passing

[alexlancaster]

It's not actually passing, if you look in the log, it's an issue in the windows job step, that I've seen before; it doesn't "fail-fast" like Linux and takes the fact that the coverage xml command passes as the whole step passing

Ah, well, makes sense. It did seem unusual that the failure would be platform dependent. So it's back to being a sphinx issue.

[AA-Turner]

114093cff01562a6a50f8c88b59c7b2fed52a39a is the first bad commit
commit 114093cff01562a6a50f8c88b59c7b2fed52a39a (HEAD, refs/bisect/bad)
Author: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Date:   Thu Jun 19 02:25:59 2025 +0100

    Use ``_parse_str_to_doctree()`` in ``Builder.read_doc()`` (#13676)

 sphinx/builders/__init__.py                        | 52 +++++++++++-----------
 sphinx/util/docutils.py                            | 16 ++++++-
 .../test_directive_object_description.py           |  2 +-
 3 files changed, 43 insertions(+), 27 deletions(-)

• Use _parse_str_to_doctree() in Builder.read_doc() sphinx-doc/sphinx#13676
• sphinx-doc/sphinx@114093c

[AA-Turner]

@chrisjsewell I believe the error is fixed with sphinx-doc/sphinx#14194.

However, I would strongly suggest not using Filter anymore here, I think it is the correct long term solution given docutils.nodes.meta is now a first-class node.

A

[chrisjsewell]

@chrisjsewell I believe the error is fixed with sphinx-doc/sphinx#14194.

Thanks! will try it out soon

However, I would strongly suggest not using Filter anymore here, I think it is the correct long term solution given docutils.nodes.meta is now a first-class node.

I didn't actually write this bit of code originally 😅 , so absolutely open to other solutions from you or anyone 👍

A

[chrisjsewell]

@AA-Turner v9.1.0rc1 no longer has the KeyError: 'writer' issue, but still fails to create the initial field_list for font-matter as before, so it looks like a new solution may still be necessary

[AA-Turner]

What's the new error?

A

[AA-Turner]

@chrisjsewell I've opened #1080 for the meta nodes.

I believe the remaining failures are caused by the tests themselves.

Sphinx has stopped using Publisher.apply_transforms(), so the monkeypatches need updating, including in sphinx_pytest. (SphinxTransformer.apply_transforms() is now used instead)

The sphinx.domains.javascript.JSConstructor failure/change seems correct (removing brackets, see the upstream PR in the JS domain). I don't know how to make a test in your custom markdown docs conditional on Sphinx version so I will leave both of these to you!

A

[Paebbels]

Is there any update on the progress?
sphinx-design#248 (pointing to this pull request) is the last missing piece for my projects to finally update to Sphinx 9.x

@chrisjsewell: I see @AA-Turner proposed a pull request: #1080

[Copilot]

Pull request overview

This PR adds support for Sphinx version 9 while maintaining backward compatibility with Sphinx 8. The changes update dependency specifications, test configurations, and test fixtures to accommodate behavioral differences between the two Sphinx versions.

Changes:

• Updated Sphinx version constraints from <9 to <10 in dependencies
• Added Sphinx 9 to the test matrix and tox environments
• Updated test fixtures and test code to handle differences in Sphinx 8 vs 9 output

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pyproject.toml	Updated Sphinx dependency constraint and pinned sphinx-pytest version
tox.ini	Added Sphinx 9 to test environments
.github/workflows/tests.yml	Added Sphinx 9 to CI test matrix and reorganized test commands
tests/test_renderers/test_fixtures_sphinx.py	Added compatibility handling for Sphinx 9 transformer and output format changes
tests/test_renderers/fixtures/sphinx_syntax_elements.md	Updated expected output for image elements with Sphinx 9 attributes
tests/test_renderers/fixtures/sphinx_roles.md	Updated expected output for JavaScript class role
tests/test_renderers/fixtures/sphinx_directives.md	Simplified skip markers by removing SPHINX4-SKIP prefix

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chrisjsewell]

nearly there, just have to fix some surfaced windows errors with snapshot tests