Support Dynamic Document Injection

[shimizukawa]

Sphinx is a great tool. However, the need to create rst documents on disk prior to running sphinx-build is a bit inflexible. Sphinx-apidoc works around this limitation by requiring a separate script execution to manage api rst files. While this works, it requires extra steps and maintenance of the on-disk rst files, and additional tools like sphinx-apidoc for each set of dynamic documents generated.

I propose an enhancement to support dynamic document/doctree or perhaps even raw rst injection during sphinx-build execution. Ideally, this would be a hook or similar mechanism that extensions could use to inject documents prior to builder processing.

---

• Bitbucket: https://bitbucket.org/birkenfeld/sphinx/issue/1139
• Originally reported by: Bob Ziuchkovski
• Originally created at: 2013-04-08T21:35:48.477

[webknjaz]

I was wondering about the same problem and tried to return a non-existent document name from the callback for env-get-outdated in hope that I'd catch it later via source-read and return some generated content. But it seems to be silently ignored and is not on the docnames list even in the next event that's happening (env-before-read-docs).
I guess Sphinx excludes invalid doc names right after that event. I'm curious if it could be preserved...

[tk0miya]

Sphinx expects that the filenames that extension returns via env-get-outdated event exist in the file system. At present, there is no way to pass the non-existent documents from extensions.

[webknjaz]

Could you point out where this check happens, please? I'd like to check how hard would it be to improve it.

[tk0miya]

Here:

sphinx/sphinx/builders/__init__.py

Lines 378 to 379 in 4c91c03

	for docs in self.events.emit('env-get-outdated', self.env, added, changed, removed):
	changed.update(set(docs) & self.env.found_docs)

The filenames given from event handlers are compared with found_docs to filter non-existent documents.

[webknjaz]

@tk0miya thanks! Would it be out of line to mutate what found_docs returns from an event handler?

[tk0miya]

I'm not sure what will be happened if this line accepts non-existent files. I guess many Sphinx components expect the file must exist. So we have to check whole of Sphinx.

[AA-Turner]

Partial Duplicate of GH-6829

Documents can be added ('injected') at the builder-inited event, which happens before Sphinx has looked for source files and found_docs is populated. We could consider adding a new event more obviously named for adding extra files, but if you would like to do so please open a new issue.

A

[webknjaz]

@AA-Turner last time I tried that Sphinx was auto-removing those from found_docs if there were no corresponding files on disk.. Has this changed? Do you have any examples?

[AA-Turner]

There's no support for entirely in-memory sources at the moment -- I had interpreted this feature request as mainly the dynamic generation aspect. I'd suggest writing to disk and then adding a second callback (either via atexit or build-finished) to delete the file.

Feel free to open a new feature request for in-memory sources if it would be useful, though it would probably be a big project.

A

[webknjaz]

@AA-Turner the OP described what you propose as working. apidoc essentially does that (short of cleanup). So I think that the only correct way to interpret it is supporting in-memory generated documents...

[AA-Turner]

Another live example of document generation is the PEPs repo, where all of the indices (including PEP 0) are written to disk during the build process.

@webknjaz would you be happy opening a new issue? I think it is slightly different, as in-memory sources don't have filenames, last-modified times, etc, so it may be more work to implement -- whereas generating documents and writing them to disk has worked for a while.

We could look to add a recipe for this to the docs as a stop-gap?

A

---

As an aside, you're listed as an owner of the @reStructuredText organisation on GH -- please could I also be made an owner? I'd like (eventually) to move the rST spec there... (though off topic now!)

[webknjaz]

We could look to add a recipe for this to the docs as a stop-gap?

If you're talking as writing to disk, that's probably well-known already.

As for in-memory files difficulties, I don't imagine Sphinx itself handling that, just not blocking the addition. I was thinking that a Sphinx extension injecting such documents would handle both generating the content and the cache invalidation. One improvement here could be having a hook point in Sphinx that would allow returning custom cache key values..

If you think it's separate, I'll try to find time to document it later.

P.S. I improved your org privileges as asked.

[AA-Turner]

(re-opening to avoid this getting auto-locked in a week)