Add an include directive that allows one filetype to be included in another #7739

eric-wieser · 2020-05-28T07:32:40Z

Feature or Bugfix

Feature

Purpose

Primarily to allow markdown files to be included within rst files (like m2r's .. mdinclude
Works for all other input parsers too

Upstreaming from pygae/galgebra#413, where this was very useful

eric-wieser · 2020-05-28T07:33:04Z

sphinx/directives/other.py

@@ -383,6 +472,7 @@ def setup(app: "Sphinx") -> Dict[str, Any]:
    directives.register_directive('hlist', HList)
    directives.register_directive('only', Only)
    directives.register_directive('include', Include)
+    directives.register_directive('includedoc', IncludeDocument)


This name is bad, would appreciate feedback on a better name

Alternative directives that better distinguish it from include might be: parse, includeother, includeparsed, etc.

eric-wieser · 2020-05-28T08:09:24Z

sphinx/directives/other.py

+
+        # copied code ends
+        app = self.env.app
+        filetype = get_filetype(app.config.source_suffix, path)


Probably makes sense to provide an option to overload this, for files with weird extensions like .inc or .md.inc etc, something like

filetype = self.options.get('filetype', None) if filetype is None: filetype = get_filetype(app.config.source_suffix, path))

+0: But nobody does not know about filetype. So we have to add it to our document.

tk0miya · 2020-05-30T04:20:20Z

-1: I don't know why such feature is needed. We already have the toctree feature to build multiple documents into the one project. In addition, merging other document is much complicated for both developers and users (path, warning, and so on).

tk0miya · 2020-05-30T04:37:46Z

doc/usage/restructuredtext/directives.rst

+   .. rst:directive:option:: start-line
+                             end-line
+                             start-after
+                             end-after


Are these filtering options needed? In my experience of literalinclude directive, many filter options are requested. So I hesitate to add filtering options.

Yes, see pygae/galgebra#413 for a useful application. It allows me to split a README.md into pieces, and include each one in a different rst document.

These filters are part of the built-in .. include, so it seems reasonable to keep them anyway.

tk0miya · 2020-05-30T04:39:33Z

sphinx/directives/other.py

+        path = relative_path(None, path)
+        path = nodes.reprunicode(path)
+        encoding = self.options.get(
+            'encoding', self.state.document.settings.input_encoding)


config.source_encoding is better for the default value.

This code is copied from the built-in ..include

tk0miya · 2020-05-30T04:41:15Z

sphinx/directives/other.py

+                                        encoding=encoding,
+                                        error_handler=e_handler)
+        except UnicodeEncodeError:
+            raise self.severe(u'Problems with "%s" directive path:\n'


u prefix is no longer needed, right?

Indeed - it's here because this code is copied directly from docutils

tk0miya · 2020-05-30T04:53:38Z

sphinx/directives/other.py

+
+        # copied code ends
+        app = self.env.app
+        filetype = get_filetype(app.config.source_suffix, path)


+0: But nobody does not know about filetype. So we have to add it to our document.

tk0miya · 2020-05-30T04:57:42Z

sphinx/directives/other.py

+        parser = app.registry.create_source_parser(app, filetype)
+
+        sub_document = new_document(path, self.state.document.settings)
+        parser.parse(rawtext, sub_document)


Why do you generate new document here? It prevents to share local reference context (ex. footnotes, hyperref, etc.).

Is there another way to invoke the parser?

I figured it didn't matter, because if the included document is in a different file it's very unlikely to reuse footnotes.

I figured it didn't matter, because if the included document is in a different file it's very unlikely to reuse footnotes.

The document object is large database for the doctree. So all references would be broken. Is it okay to conflict node_IDs between parent and included document? I guess :ref: role for included document may not work also.

Is there another way to invoke the parser?

How about parser.parse(rawtext, self.state.document) instead?

Does docutils allow a document to be parsed into multiple times? How does it know where the current insertion point is?

sphinx/directives/other.py

…nother Most usefully, this allows embedding markdown within rst.

doc/usage/restructuredtext/directives.rst

RabidCicada · 2020-08-07T17:12:23Z

Holy cow @eric-wieser . Thankyou for your work on this.

I currently have a code base that is heavily locally documented in each subdirectory with README.md's about each component authored by developers that are much more comfortable with Markdown.

I have crafted my system to parse rst, md, plantuml, and auto harvest C++ API docs using doxygen. After following #701 and looking for a good solution...I've settled on replicating my entire folder structure under docs and using softlinks so that relative linking works both natively on gitlab/github with the Markdown documents but also when built by sphinx.

While this does not solve that problem I appreciate your work in contributing something that would be very useful.

RabidCicada · 2020-08-07T17:20:23Z

This might partially solve relative linking instead of relying on softlinks. So then it would work on Windows. Any thoughts about if this could take the place of softlinks. I think your .. includeDoc:: in README.rst's with recommonmark could replace my softlink approach.

docs/
    src/
        folder1/
            README.md <softlink>
        folder2/
            README.md <softlink>
        folder3/
            folder4/
                README.md <softlink>
        README.md <softlink>
        index.rst
        conf.py
    build/
    Makefile
folder1/
    README.md
folder2/
    README.md
folder3/
    folder4/
        README.md
README.md

RabidCicada · 2020-08-07T17:33:06Z

I also have looked at and followed a lot of the disussion here: #7000. similar issues.

I like your solution here as one that might actually be mergable since while @tk0miya has suggested he supports the idea all attempts so far have been met with only negative criticism of the solution and resistance to changing deep parts of sphinx ( #2354 and #7913). Not that the criticism isn't warranted or that the guardianship of the code base isn't a good thing.

@tk0miya is there a chance that you can:

suggest a positive path forward for implementing a solution that will be acceptable to you for this MR?

The approach used by recommonmark's own documentation is to include a symbolic link file inside the `docs` directory, instead of trying to do a `include`. References: readthedocs/recommonmark#191 sphinx-doc/sphinx#701 sphinx-doc/sphinx#7739

eric-wieser commented May 28, 2020

View reviewed changes

eric-wieser force-pushed the include-any branch from 79e55a7 to 80e5b63 Compare May 28, 2020 07:41

eric-wieser mentioned this pull request May 28, 2020

..include:: statement doesn't parse markdown files correctly #2840

Closed

eric-wieser force-pushed the include-any branch from 80e5b63 to a24eba2 Compare May 28, 2020 08:04

eric-wieser commented May 28, 2020

View reviewed changes

eric-wieser force-pushed the include-any branch from a24eba2 to 2cd52a9 Compare May 28, 2020 17:47

tk0miya added markup type:proposal a feature suggestion labels May 30, 2020

tk0miya reviewed May 30, 2020

View reviewed changes

Add an include directive that allows one filetype to be included in a…

cc3cca2

…nother Most usefully, this allows embedding markdown within rst.

eric-wieser force-pushed the include-any branch from 2cd52a9 to cc3cca2 Compare May 30, 2020 07:56

eric-wieser commented Jun 8, 2020

View reviewed changes

doc/usage/restructuredtext/directives.rst Outdated Show resolved Hide resolved

eric-wieser added 2 commits June 8, 2020 17:48

Update doc/usage/restructuredtext/directives.rst

478108f

Update sphinx/directives/other.py

dbc43e3

DominiqueMakowski mentioned this pull request Jun 20, 2020

Enable Markdown files to be added to the Sphinx docs neuropsychology/NeuroKit#275

Closed

abravalheri mentioned this pull request Jan 6, 2021

Workaround to include relative readme into docs directory pyscaffold/pyscaffoldext-markdown#6

Merged

tk0miya closed this May 9, 2021

tk0miya deleted the branch sphinx-doc:3.x May 9, 2021 03:24

github-actions bot locked as resolved and limited conversation to collaborators Jul 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an include directive that allows one filetype to be included in another #7739

Add an include directive that allows one filetype to be included in another #7739

eric-wieser commented May 28, 2020 •

edited

Loading

eric-wieser May 28, 2020 •

edited

Loading

rpanderson Jun 9, 2020 •

edited

Loading

eric-wieser May 28, 2020 •

edited

Loading

tk0miya May 30, 2020

tk0miya commented May 30, 2020

tk0miya May 30, 2020

eric-wieser May 30, 2020 •

edited

Loading

tk0miya May 30, 2020

eric-wieser May 30, 2020

tk0miya May 30, 2020

eric-wieser May 30, 2020

tk0miya May 30, 2020

tk0miya May 30, 2020

eric-wieser May 30, 2020

tk0miya May 30, 2020

eric-wieser Jun 8, 2020

RabidCicada commented Aug 7, 2020

RabidCicada commented Aug 7, 2020 •

edited

Loading

RabidCicada commented Aug 7, 2020

Add an include directive that allows one filetype to be included in another #7739

Add an include directive that allows one filetype to be included in another #7739

Conversation

eric-wieser commented May 28, 2020 • edited Loading

Feature or Bugfix

Purpose

eric-wieser May 28, 2020 • edited Loading

Choose a reason for hiding this comment

rpanderson Jun 9, 2020 • edited Loading

Choose a reason for hiding this comment

eric-wieser May 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tk0miya commented May 30, 2020

Choose a reason for hiding this comment

eric-wieser May 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RabidCicada commented Aug 7, 2020

RabidCicada commented Aug 7, 2020 • edited Loading

RabidCicada commented Aug 7, 2020

eric-wieser commented May 28, 2020 •

edited

Loading

eric-wieser May 28, 2020 •

edited

Loading

rpanderson Jun 9, 2020 •

edited

Loading

eric-wieser May 28, 2020 •

edited

Loading

eric-wieser May 30, 2020 •

edited

Loading

RabidCicada commented Aug 7, 2020 •

edited

Loading