New Sphinx tutorial for beginners

[astrojuanlu]

tl;dr: We are proposing a new Sphinx tutorial for beginners (no rush! we know that you're busy with the upcoming release) and would like to know people's thoughts.

Hello Sphinx developers! As part of the Chan Zuckerberg Initiative grant that Read the Docs received last year, one of the key tasks we are focusing on is doing "advocacy work around documentation in the scientific community", which is one of my duties as Developer Advocate.

In particular, after talking to lots of users from a variety of backgrounds, we identified a gap in the Sphinx documentation: although it is super detailed and works great as reference material, many people seemed to agree that it would be useful to have a good beginners tutorial. As a result, we have been iterating for a few days and we would like to propose adding a new beginners tutorial to Sphinx.

We commit to writing it incrementally (instead of sending one huge pull request after 6 months), maintaining it, and keeping it up to date, while using this opportunity to contribute other documentation fixes and new features along the way, if needed. We would like this tutorial to be The Best™ resource to direct people willing to learn how to author their technical documentation with Sphinx.

Here you can find the rendered version of the contents we are proposing https://docs--8106.org.readthedocs.build/en/8106/development/design/new-sphinx-guides.html#sphinx-tutorial (and here is the originating pull request in case you want to add comments there directly - the fact that it's on the RTD repository was only a way so we could easily iterate over it "internally"). Summarized, these would be:

1. The Sphinx way
2. About this tutorial
3. Getting started
4. First steps to document our project using Sphinx
5. Customizing Sphinx configuration
6. Writing narrative documentation with Sphinx
7. Describing code in Sphinx
8. Autogenerating documentation from code in Sphinx
9. Deploying a Sphinx project online
10. Appendix: Using Jupyter notebooks inside Sphinx
11. Appendix: Understanding the docutils document tree
12. Appendix: Where to go from here

In particular, we'd like to:

• Be as pedagogical as possible, to best serve beginners.
• Conceive a linear "learning path" that people can follow so they can learn the fundamentals step by step.
• Write the tutorial with a generic technical user in mind, while including some appendixes that can be useful for scientists (like integrating math formulas and Jupyter notebooks).
• Use Markdown all throughout the tutorial, thanks to MyST-Parser.

Pinging @ericholscher from RTD, @choldgraf and @chrisjsewell from MyST, @melissawm who is doing a fantastic job with the NumPy tutorials, @pradyunsg who has been working on Sphinx a lot recently, @laisbsc and @emilopez who gave me initial feedback about some of these ideas, and @lornajane who also expressed interest about this in the Write the Docs Slack workspace.

What are your thoughts @tk0miya @shimizukawa?

[chrisjsewell]

Quick initial thought is yeh that all sounds great thanks, and definitely on-board 👍

With our jupyter-book documentation, @choldgraf has recently been working to improve the initial tutorial: https://jupyterbook.org/start/your-first-book.html, which is pretty much "how to create a sphinx project, without knowing you are using sphinx" and, as such, has a lot of synergy with your proposal.

I'd also not that, in parallel to that, we have been looking to extract the "bespoke" code in jupyter-book, to the point that it is basically a pre-defined set of sphinx extensions and configurations.
Two of the key aspects are:

• https://github.com/executablebooks/sphinx-external-toc, which is now out of beta and allows the option to define your site map in a single YAML file (rather than via toctree directives).
• Add static configuration (Sphinx.toml) #9040 to allow configuration to be defined in a YAML file.

Both of these we have found to be very helpful for beginners and make it possible, for example to generate a basic sphinx site from a set of Markdown files without requiring any modification to them

[melissawm]

Hi @astrojuanlu - this looks really good! I will also ping @rossbar who has been working on the NumPy docs team and might want to share his thoughts as well.

I'd be happy to help too - this is something that would definitely help our contributors. Thank you for working on this!

[jakobandersen]

Improving the documentation is always good, and the overall plan I think looks good.
As my primary interest is in

7. Describing code in Sphinx

the following is from that perspective.

• Use Markdown all throughout the tutorial, thanks to MyST-Parser.

I haven't followed the markdown side, but I find this point a non-trivial decision. If new users read this tutorial and get used to MyST flavoured Markdown syntax, but then search for help online, wouldn't they find almost only reST code? While reST has some problems it seems icky to have two different formats in the Sphinx documentation (unless everything gets rewritten to MyST).

As for MyST it self, I looked at the intro but I didn't find an explicit answer to how one nests directives. Though in the syntax section there is some more info.
For the example

.. cpp:class:: template<typename T> Vector

   .. cpp:class:: iterator

      .. cpp:type:: value_type = T

here the nesting has semantic meaning, and the (forced) indentation in source code makes it readable.
For MyST it looks like this nesting can not be preserved in the source code and the best will be something like

`````cpp:class:: template<typename T> Vector

   ````cpp:class:: iterator

   ```cpp:type:: value_type = T
   >```
   >````
>`````

(with the > in the last three lines removed, otherwise Github would get confused)
Is that correct? I'm not sure I understand how to do arbitrary nesting, and I would strongly prefer something were the nesting is indicated clearly in the source, even when you not fit the complete source of the outer block in a single screen (e.g., indentation for each level).
I probably don't understand how directives work in MyST, but it would be nice to see non-trivial examples.

[pradyunsg]

https://pradyunsg.me/furo/reference/ has a bunch of examples of MyST vs reStructuredText.

I've personally also recommended MyST in https://pradyunsg.me/furo/recommendations/, since Markdown is more popular as a markup language. That said, I think for the Sphinx tutorial, it might make sense to either keep it as only reStructuredText. Or to have reStructuredText as default with a tab to switch to MyST like the reference documentation linked above.

The way to do nesting in MyST is increasing backticks. So, basically your example is almost correct -- there's no indentation necessary.

[jdillard]

Use Markdown all throughout the tutorial, thanks to MyST-Parser.

I strongly prefer a particular extension not be required for beginners to learn how the core of Sphinx works. I understand your grant is for advocacy work around documentation in the scientific community and that extension is more heavily used there, but I would prefer that preference not be forced on all new users.

[pradyunsg]

I guess the specific pages I wanna point to for showcasing the differences between the two markup languages would be:

• Nesting example: https://pradyunsg.me/furo/reference/admonitions/#nesting-admonitions
• MyST has a much easier to mentally model approach to hyperlinks: https://pradyunsg.me/furo/reference/hyperlinks/

[pradyunsg]

Stepping away from the markup language choice, which feels a bit flame-war-like, I do like how the tutorial is structured.

A couple of things that look weird though:

Appendix: Understanding the docutils document tree

Hmm, is this about what the toctree produces? I feel like this should immediately follow the section on narrative documentation, instead of being delegated to an optional appendix. It's fairly important to understand this, to be able to write or work with Sphinx documentation IMO.

---

Intersphinx is noticeably missing from this, and AFAIK, that's basically a big reason that folks still continue to choose Sphinx for documentation despite all the alternative documentation genrrators available. In other words, it's a unique selling point for Sphinx, and IMO we should at least have an appendix about it to introduce it to someone who hasn't heard of it.

[jakobandersen]

@chrisjsewell, Indeed, without indentation it is clear, but it is mentioned that one can use a single level of indentation. @pradyunsg admonition example shows the same. While for that case multi-level nesting seems dubious, it is inherently a part code documentation, at least for C++ and C.
I'm not at all convinced this is a good syntax.

@pradyunsg, it's a very nice site to see the differences between the two input formats! (though I noticed that for tables the MyST example is missing tables with row and column spanning).

[pradyunsg]

@panhaoyu,

Oops, wrong mention. ;)

(though I noticed that for tables the MyST example is missing tables with row and column spanning)

It probably has something for that. I had gotten lazy by the time I wrote that, so the later pages of that whole section aren't fully polished. 😅

[astrojuanlu]

Whoa! Didn't expect so many comments :) Quickly answering a couple of easy ones:

Appendix: Understanding the docutils document tree

I referred to this: https://docutils.sourceforge.io/docs/ref/doctree.html so, more like the internals of docutils rather than the toctree itself. Which perhaps we should treat more in depth?

Intersphinx is noticeably missing from this, and AFAIK, ...

It's a super good point, thanks @pradyunsg!

[astrojuanlu]

About the markup choice: regardless of which one we use, Sphinx is much more than that. I would happily discuss the merits of MyST vs reST, perhaps at a later stage, since I don't think it affects the contents of the tutorial itself.

[ericholscher]

Also of note, we aren't trying to write tutorials for every feature of Sphinx. We want to keep this sized to something that people will actually do, instead of adding every possible feature. We should include more tutorial-style content in the Sphinx docs, but we should be pretty focused and concise on what actually makes it into the official tutorial.

I love this section from Jacob's blog post long ago that covers what to include in the tutorial: https://jacobian.org/2009/nov/10/what-to-write/#tutorials -- something to keep in mind here.

I think a way to think about it might be "What are the three most unique and valuable Sphinx features that we must be sure to include, to make sure users get the most 'this is awesome' from the tutorial"

I think my list is probably:

• Cross-references & Intersphinx
• Includes (including literal-include)
• Mixing of auto-generated & narrative content
• csv-table (mostly kidding, but its huge)

[chrisjsewell]

I love this section from Jacob's blog post long ago that covers what to include in the tutorial

FYI I find this a very helpful guide for documentation writing: https://documentation.divio.com/

[astrojuanlu]

It has a new home and a classier name :) https://diataxis.fr/

[ericholscher]

Yep -- those share a direct lineage to Jacob's blog post back in 2009 via the Django docs -- a story for another day tho :)

[choldgraf]

I'm not going to dive into the specific markup language etc stuff, but here are a few thoughts:

• I think this is great idea, and fills a much-needed gap in documentation
• The most important thing is to consider what person this documentation is meant for, and what assumptions we can make about them in order to maximize the likelihood that they'll finish the tutorial with a good foundation and wanting to learn more.
• I agree the diataxis framework is an excellent guide to help this effort (as well as a good guide to decide which things need to live in appendices vs. in-line with the tutorial)
• Remember that the goal of a tutorial is not to teach somebody everything, but to get them started and give them a taste of what they could do if they learned more. :-)

And maybe more generally - I think it'd be helpful to provide a place for tackling questions and feedback that wasn't in one gigantic issue in the sphinx-doc repo :-) is there either a GitHub Discussion, or issues in another repository, where more specific conversations could happen?

[webknjaz]

I like the idea and the general outline. I also agree that having intersphinx well-explained would be great.
I think that it's preferable to keep using RST (per many explanations above and for the consistency with other docs) but it would be a nice thing to have if you added a small section/tip on "by the way, you can mix and match different markups on the same site, here's how you do this with MyST". Mixing human-written vs auto-generated content is a nice topic too.

re:includes — pleeeeeease add examples of "ends-before"/"start-at" with "emphasize" for snippet inclusion.

[tk0miya]

Great idea! It must be a missing piece of Sphinx!

Appendix: Understanding the docutils document tree

I don't know this is needed for tutorial. In my understanding, doctree is an internal data representation. So the main users of doctree are developers, I think.

Use Markdown all throughout the tutorial, thanks to MyST-Parser.

I feel a bit strange the official Sphinx tutorial depends on 3rd party extension. I know markdown is much popular. So how about introducing MyST-Parser as an appendix?

[choldgraf]

My thinking on a markup language and the role of a tutorial:

• Tutorials should make a set of assumptions about a learner, and define a set of core learning goals that we want the learner to come away with.
• The core goals must be decided upon so that they're realistic, useful, and most importantly so that they don't frustrate the learner. The goal of a
• The question about "what markup language should be used" should be driven by these two points, rather than by technical details like "what is core vs. 3rd party extension"

So I guess another way of re-framing the markup language question is:

Do you believe that reStructuredText is a fundamental part of Sphinx, such that a description of "Learn how to use Sphinx" must crucially also depend on "Learn how to write reStructuredText"?

Alternatively, do you believe that any particular markup language is not a "crucial part of Sphinx", in which case the question is then "which markup language will help the learner the most, reduce cognitive load, and be most useful for their likely use-case".

And finally: I think for the kind of learner this tutorial is designed for, we should assume that they are familiar with Markdown but not reStructuredText. In my experience (esp. for the scientific crowd) people know reStructuredText because they use Sphinx, and thus those people are not what this tutorial is designed for.

[jdillard]

Do you believe that reStructuredText is a fundamental part of Sphinx

I think reStructuredText (specifically :ref:, :doc:, .. include:: and the like) is a fundamental part of what differentiates Sphinx. I think the how to use sphinx can have the byproduct of also showing why you should use Sphinx (assuming it fits the readers use case and they can see that). Otherwise this just turns into another SSG tutorial, which I think is one of problems when someone who is new to SSGs makes the mistake of lumping Sphinx into the same group as the other popular SSGs that mainly support markdown (there is some AsciiDoc support as well usually, but I'm not familiar with how much).

[jdillard]

I should clarify that I don't think MyRST should be used for any of the tutorials. I'm more concerned with how The Sphinx Way portion is worded, but I just read the linked pull request rendering (oops) and it seems that was covered.

[tk0miya]

I'm afraid of the reader of the tutorial posts a question or bug reports because I've never used it and will not support it. Do you know the MyST-parser supports the full features of Sphinx?

Err, trying to resist the 🔥 lol; roles and directives (e.g. ref and include) are indeed a strong feature of docutils/sphinx, but they are not just the preserve of reStructuredText, you can use any of them in MyST as well (and essentially you could write any source syntax format to support them).

off topic: IMO, it's "reST Flavored" markdown. It must be useful, but not a commonly used format. So I hesitate to call it "Markdown".

[humitos]

I'm big 👍🏼 on having a new tutorial since it's something that I've been asked hundred times, and I think it's needed.

That said, I wanted to chime in to give a completely personal opinion here about the markdown/rst choice. I agree with @choldgraf that defining the "goal of the tutorial" is a crucial part and I think that the expected audience is also important.

I think it's good if the reader immediately discovers that the tutorial can be done using reST and/or MyST. This is important to me to not loose people that come with some knowledge about Markdown (probably because they use it a lot on GitHub and they are already used to). I think that if the tutorial presents everything only reST and says nothing about it can be done with Markdown as well, we will probably loose that user.

I think that we will all benefit from presenting both options here. I don't think it has to be one or the other. In particular because the content of the tutorial shouldn't be different if we decide only reST or only MyST. I assume that only the code examples will be different. To do this, I like the idea of using tabs that @pradyunsg showed, where you can see all the page in reST or MyST just by clicking a tab.

I'd also say that if there are limitation on these flavors, it seems useful to highlight them at some point in the tutorial before you decide the flavor for your documentation. This could be some notes and highlights that make the user to decide which flavor is best for their use case.

[tk0miya]

It's tested against every built-in docutils/sphinx role and directive:

It does not mean covering all features of Sphinx. How about i18n? How about autodoc and autosummary? To become the first-class parser of Sphinx, we need to test the combination with the built-in features. Or, users will get confused on troubling. "I followed the tutorial. But it's a trap. There are some not working features for Markdown..."

I found this topic on MyST-paser: https://myst-parser.readthedocs.io/en/latest/using/howto.html#use-sphinx-ext-autodoc-in-markdown-files
If my understanding correct, users should know what feature is supporting markdown, and another is not on writing document, right? I think we need to describe it on our document too, not only on the tutorial.

In other words, the hidden topic of this issue would be to make MyST-Parser bundled, I think. I agree it's enough worthy. But many times and resources are needed to realize the integration. So it should be another topic, IMO.

[jakobandersen]

Regarding the tutorial structure, regardless of input format:
I don't think the basic tutorial should get into things like intersphinx, but it would be good to have an appendix as a "how to ...". Of other bundled extensions for how-to appendices I think sphinx.ext.todo and sphinx.ext.mathjax might be good to cover.

For the code documentation part:
I think it would be interesting to make part of this a kind of tabbed version, to show-case different domains. The concepts are basically the same across domains, but different languages have different audiences and for the beginner it would get rid of the mental overhead of translating between programming languages. I can help with the C and C++ part of this if needed.
In relation to this, for the automatic code generation it may be worth mentioning Breathe as an extension for doing similar things as autodoc.

Regarding format (sorry for the "flame war" but this is really core to an official tutorial):
Having a double-tabbed version with reST as the primary might be ok, but it is still icky if an official Sphinx tutorial describes MyST when Sphinx effectively have no support for it, and as @tk0miya just showed, there are some limitations.
Perhaps we should take the discussion of MyST somewhere else, but for now:
Both reST and Markdown supports basic document formatting, and indeed reST has a core flaw in that nested inline markup is not supported. However, from the perspective of writing API documentation, the core of Sphinx is to me the use of roles and directives, with nesting. If I understand correctly then Markdown does not have these concepts, hence MyST having to invent what looks like a crutch to let people write them anyway. If the benefit of MyST is that people might know basic Markdown from somewhere else, does it really help them much when roles and directives require non-standard syntax anyway? Is it too late for MyST to learn a better syntax for directives?

[chrisjsewell]

I don't think the basic tutorial should get into things like intersphinx
or the automatic code generation it may be worth mentioning Breathe as an extension for doing similar things as autodoc.
from the perspective of writing API documentation

I feel you have a very different take on this tutorial to me 😬
I think intersphinx is exactly what we should be mentioning, and really I would set-aside anything to do with API documentation, like autodoc or breathe, to an appendix, since I think that is more of a specialist concern

[jakobandersen]

I think intersphinx is exactly what we should be mentioning

I have mostly seen it in connection with API documentation and the current docs also mentions it in that context. It is of course a general mechanism for foreign project lookup, but I think it is beneficial to have seen how to use domains first, so get the most out of intersphinx. Perhaps I'm missing some context of how intersphinx is used in different ways?

and really I would set-aside anything to do with API documentation, like autodoc or breathe, to an appendix, since I think that is more of a specialist concern

There are at least two levels to API documentation: the manual layer where you directly use a domain to write you declarations with directives, and then the automatic layer where both autodoc and Breathe lives. I think the manual layer should be part of the base tutorial, and the automatic should probably be a prominent appendix, with autodoc for the Python domain. I do not suggest Breathe being more than a brief mention to answer the question of "I don't document Python code, what are my options?".

[stevepiercy]

8. Autogenerating documentation from code in Sphinx

I monitor the two tags python-sphinx and restructuredtext on StackOverflow. I can affirm that the number one frequently asked question is, "Why doesn't my project's API documentation build?". I would strongly support its inclusion so that I can point users to this documentation, as the Sphinx documentation is not altogether clear. I find that the RTD documentation in its FAQ is more helpful.

[humitos]

Do folks have other audiences in mind that we should consider while creating the tutorial content?

Translators. In my experience working on the CPython Spanish documentation we do have a lot of people that are not developers and are not connected with technical writing at all. The translation is their first contact with FLOSS. We started teaching them the basics of git (to clone the repo), poedit (to edit PO files), some reST syntax (bold, italic, quotes, etc), and very little of Sphinx (to understand how to translate roles and how to read Sphinx's build error messages from the CI).

[webknjaz]

I can affirm that the number one frequently asked question is, "Why doesn't my project's API documentation build?". I would strongly support its inclusion so that I can point users to this documentation, as the Sphinx documentation is not altogether clear.

I'd also add a pointer to https://github.com/sphinx-contrib/apidoc which allows continuous apidoc generation VS one-time rst file creation that vanilla sphinx does.

[astrojuanlu]

I monitor the two tags python-sphinx and restructuredtext on StackOverflow. I can affirm that the number one frequently asked question is, "Why doesn't my project's API documentation build?"

Thanks for the pointers @stevepiercy, I'll start monitoring those as well.

[pradyunsg]

very little of Sphinx (to understand how to translate roles and how to read Sphinx's build error messages from the CI)

This sounds like we need a separate translation tutorial.

[pradyunsg]

Okie, I think the next step is figuring out what these audiences might want an answer to when they're new to Sphinx.

Based on the audiences we identified here so far, I've gone ahead and created a collaboratively editable pad for doing this: https://hackmd.io/@pradyunsg/HkX88e4uO

Please feel free to drop inline comments there, or to add/edit the questions there.

[choldgraf]

@pradyunsg I like that breakdown. IMO I wouldn't actually change much from what you've got.

I think that the "common" steps in your write up are a good scope for an "intro to sphinx" tutorial described here.

I think that the other more user archetype-specific points are great topics for follow-up explanations or other more focused tutorials. (Eg I think there should be a separate tutorial that is specific to internationalization and translation, and there should also be another tutorial that was called "Getting started with markdown for Sphinx" and is hosted on the myst markdown documentation, at least until the sphinx community is comfortable hosting a tutorial like that on its own docs)

[astrojuanlu]

The conversation seems to have settled down a bit, so I would like to thank everyone again for your input! @pradyunsg document with questions that need to be addressed (both for a generic audience and for more specific users) looks great to me, and I took note of the suggestions to our initial table of contents. I agree with @choldgraf and others about keeping the beginners tutorial focused and leaving the door open for more specialized ones in the future.

We will start step by step, so expect a pull request in a few weeks with the first sections, so we can also find a good place and navigation for it.

[rgacote]

I'd appreciate seen an iterative approach to a final site. Once you get through (frequently very basic) tutorials you're dumped into the terse and detailed technical docs.

Perhaps view these as individual modules.

• Get your site working
  • Basic rst syntax
• Select a theme
• Publish to read the docs or elsewhere
• In depth look at building tables.
• Adding images and figures, and why choose one over the other
• Todo
• Intersphinx
• Checking links in your docs
• Generating documentation.
  • Different styles like Google Doc style (Napoleon extension)
• You can also use Markdown!
• Building images with graphvix
• Fancy math (imgmath, mathjax, etc.)

[astrojuanlu]

Hi everyone, I sent a first pull request with some minimal (but complete) content on #9276, to set the initial tone and structure of the tutorial. Hope this will be the first of many to come 💪🏽

[astrojuanlu]

For the curious, here is part II: #9355

[astrojuanlu]

Hi all! The tutorial is largely complete, see #9276, #9355, #9534, and #9852 🚀 Almost all of it is visible now on https://www.sphinx-doc.org/en/master/tutorial/index.html (it needs one more merge from 4.x to master).

I wanted to add a section on intersphinx in #9424, but after some discussion, we decided to postpone it. I'm looking forward to see the new :external: role merged (#9822) and I'll be happy to add it to the corresponding section when it's available.

The only remaining open questions are about the appendixes:

• "Using Jupyter notebooks inside Sphinx" would largely be https://docs.readthedocs.io/en/stable/guides/jupyter.html, happy to discuss with the Sphinx maintainers if they would accept such guide here.
• "Understanding the docutils document tree" is probably too ambitious for a beginners tutorial, and would be most useful for extension writers, rather than documentation writers. We can talk at some other moment about expanding and updating the developer documentation, in line with some of the conversations at sphinx.testing is undocumented #7008 and other issues.
• "Where to go from here" should be a small section at the end of the tutorial, mostly with links. I'll send a follow-up PR in the coming days.

Since we purposedly pinged many people in this issue to shape the original proposal and it has already been implemented almost entirely, I propose @tk0miya @shimizukawa @jakobandersen @jfbu we agree on the remaining questions and we close this issue in favor of smaller ones.

[ericholscher]

Thanks for your hard work on this @astrojuanlu! This has been a long-standing wish of mine, and it's super exciting to be able to see it shipped! 🎉

[tk0miya]

Now I merged the 4.x branch to the master now. Your excellent work will be applied to our site. Great work!

[choldgraf]

Thanks @astrojuanlu - I hope that this helps more people get started quickly with the Sphinx ecosystem!

[tk0miya]

"Using Jupyter notebooks inside Sphinx" would largely be https://docs.readthedocs.io/en/stable/guides/jupyter.html, happy to discuss with the Sphinx maintainers if they would accept such guide here.

I think using jupyter notebook becomes one of the useful use-cases of sphinx now. So it would be better to introduce or mention it.

"Understanding the docutils document tree" is probably too ambitious for a beginners tutorial, and would be most useful for extension writers, rather than documentation writers. We can talk at some other moment about expanding and updating the developer documentation, in line with some of the conversations at sphinx.testing is undocumented #7008 and other issues.

How about adding both of them into the extension's tutorial? It's important knowledge for extension developers.
https://www.sphinx-doc.org/en/master/development/tutorials/index.html

"Where to go from here" should be a small section at the end of the tutorial, mostly with links. I'll send a follow-up PR in the coming days.

I'm looking forward to review it :-)

[astrojuanlu]

The tutorial could of course be improved and enhanced but do Sphinx devs think it's a good moment to close this issue, or rename it to "add appendixes to Sphinx tutorial"?

[astrojuanlu]

Taking the liberty of closing this issue. Thanks everyone!