support for glob: in the TOC

[parmentelat]

Proposal

Hi community !

I am a long time jupyter-book user; I wish to take this opportunity to say how grateful I am for this wonderful tool :)

One thing I am struggling with though is the TOC structure, that iiuc was mostly inherited from sphinx; in any case I still do feel a little awkward about that, like I believe quite a few other users do, if I judge from various opinions that I've read here and there ;)

For this reason mostly, I am currently giving a shot at mystmd as a possible replacement, targetting ideally the beginning of next semester, end of august;
with as primary objective the ability to define a more natural global toc, esp. more freely define collapsible areas and content-less labels

as a first step I just ran myst in a jb repo as-is, and have immediately run into the lack of support for glob: entries in _toc.yml

although I can work around that by manually expansing my globs, it is my feeling that native support for such entries by mystmd would greatly help other users who follow a similar path (in particular, I also ran into that when using the online tool that converts a github repo, sorry I can't find that url again at the moment...)

also, I am just digging into this project for the first time, but have noticed you have recently decided to go for your own toc-definition format, and in this respect I thought it might be valuable to take this into account at an early stage ?

Additional notes

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[agoose77]

Hi @parmentelat, thanks for your report!

We're actually in the process (c.f. #1188) of implementing a new TOC structure in MyST. I'm glad that you observe the need for content-less sections; that's a limitation posed by the existing Sphinx-derived Jupyter Book TOC.

Once #1188 lands, it will be possible to define a table of contents in your myst.yml. The docs for this new TOC can be seen in that PR here

[parmentelat]

Hi @agoose77
thanks for you feedback
I had noticed that move, and actually it is part of my motivation to migrate, would be a relief indeed to have a decent formalism for tocs

however I'd like to stress that my initial motivation is to a large extent orthogonal to this new track, I was more outlining a steep step in the migration process for those who have a large number of glob: entries in their toc..

[agoose77]

@parmentelat the new TOC is related to this: we will focus our efforts on a migration from JB to MyST (or JB 2, see https://executablebooks.org/en/latest/blog/2024-05-20-jupyter-book-myst/). We do still plan to support _toc.yml but my guess is that is for existing MyST-MD users who've been having success already with it. For users that are unable to use _toc.yml right now, the migration pathway seems more sensible. Though, I'll ping @rowanc1 to sense-check this opinion! (He is not in-office right now, so bear with me).

[parmentelat]

thanks for your feedback, looking forward to his insights

[choldgraf]

Just a quick question: I thought that the new TOC work does support glob-like patterns, e.g. so yo ucould do:

toc:
  - file: section-*.md

It seemed there was blob-like logic in the PR:

https://github.com/executablebooks/mystmd/pull/1188/files#diff-9d43afdfd1cd05cc3c9c95a41f41045c9de1033802e28f803d6a5be176d5a7bfR54-R76

Am I wrong there?

[agoose77]

@choldgraf yes, we support a pattern entry for glob patterns!

[choldgraf]

Ah - so in that case, @parmentelat's concern here:

as a first step I just ran myst in a jb repo as-is, and have immediately run into the lack of support for glob: entries in _toc.yml

Is actually resolved by: This functionality is in mystmd, it just doesn't require the :glob: argument. So the problem is one of "converting _toc.yml syntax into myst.yml syntax", rather than "functionality is missing". Is that right?

[agoose77]

It occurred to me after reading your comment that I never surfaced that implicit point!

Yes, MyST-MD is about to get glob support in its internal TOC. My feeling is that we deprecate the _toc.yml support where it's at, rather than fixing it, although it wouldn't be too hard to do.

[parmentelat]

thanks @agoose77 and @choldgraf for spelling it out, this implicit thing had totally escaped me so far

so let me try to summarize, so I can make sure I got it right
imagine a user shows up with a project that has a _toc.yml from jb1, with globs in it
you're saying that

• running myst on that repo as-is will simply ignore the contents mentioned in the glob
• for the contents to appear, they must somehow translate their _toc.yml into the new toc-oriented chunk as part of myst.yaml - or whatever it is that the new config file is called
• I take it you are currently designing tools for helping this stage ?
• and you're saying that the globbing feature is then supported; it's unclear to me at this point, whether it's through a file tag as @choldgraf suggests, or through a dedicated pattern tag as @agoose77 suggests
  (I can see the value of using a single name (if the former), but imho if the latter then why not use glob that people have been used to for quite some time already; but that's minor detail of course)

is that a fair description ?

[agoose77]

I take it you are currently designing tools for helping this stage ?

@parmentelat a working idea for JB 2 is:

1. jupyter book is just an alias of myst.
2. jupyter book will override myst's attempt to support _toc.yml (throwing an error - "you need to ugprade", run jupyter book upgrade ...)
3. The semi-automatic upgrade tool will replace _config.yml and _toc.yml with a new myst.yml.

whether it's through a file tag as @choldgraf suggests, or through a dedicated pattern tag as @agoose77 suggests

Right now, it's pattern. Given that the upgrade is automatic, and this new TOC format is different to JB, I think the pattern wording is slightly clearer to new users.

[parmentelat]

hey @agoose77
thanks for the clarification; it's true that picking pattern or glob does not matter much...

you've lost me a bit though, when you said

jupyter book is just an alias of myst

this seems like a rather radical statement to me; for one thing, some people like developers, who used autodoc and similar features, are going to want to stick to the sphinx-based stack; plus, jb and myst have different subcommands and usage... so it looks more like the 2 commands will coexist and people will have to opt in; were you using that in a manner of speaking, or did you mean it literally ?

[agoose77]

Kinds of "JB User"

We're still working out the UX, so I'll start by saying this all might change!

Practically speaking, the EB organisation services many different kinds of users:

1. Developers writing API docs
2. Researchers writing tutorials / sharing their work
3. Information-building communities writing guidance and best-practice

The big distinction for me is whether Jupyter Book is the entry-point to their book, or whether it is used in terms of its components, i.e. is there a conf.py or _config.yml.

We originally built upon Sphinx because it was well-established as a documentation tool, which is why the components that comprise Jupyter Book are so well-suited to (1) developers writing API docs. However, after years of exploring the problem-space occupied by Sphinx, it has become clear that Sphinx is now a limiting factor both for users and developers.

It follows that, as outlined in our blog post, we're building Jupyter Book 2 on top of MyST-MD instead of Sphinx. For users, we feel that MyST provides a better UX than Sphinx, and for developers, MyST-MD is built from the ground-up around scientific writing and technical communication with modern web-first principles.

In order to make the separation between our Sphinx work and the new MyST-MD/Jupyter Book projects clearer, we plan to migrate the new MyST efforts to a Jupyter subproject jupyter-book, such that executablebooks can focus on the existing Sphinx stack.

Exploring JB on top of MyST

When I write that

jupyter book is just an alias of myst,

I should elaborate.

Much of the existing jupyter book (1) CLI is a thin abstraction around Sphinx, for example:

• The --builder command just chooses which Sphinx builder is in use.
• The config subcommand allows you to export a Sphinx config

Anyone authoring PDFs, or wanting to add more functionality to their books ultimately has seen this leaky abstraction, and probably has a reasonable amount of Sphinx-specific (i.e. not Jupyter Book) configuration in their _config.yml. As such, we haven't been able to protect users from the complexity of Sphinx, and have required our users to learn multiple tools.

You could ask "why didn't you just use Sphinx directly?", and the answer is that it's complex; it's old, with a long history, and it does many things. It doesn't make for a very beginner/non-technical-user friendly tool as a result.

We feel that MyST is a much simpler tool to use than Sphinx. Because it's a younger project, we've been able to take what we learned from Jupyter Book and apply it from the beginning. It's designed around rich scientific communication, so the features required to support that were added at its core. Unlike Sphinx, MyST is developed on GitHub with the hope that we'll continue to grow our contributor base according to the Executable Books / Jupyter principles.

some people like developers, who used autodoc and similar features, are going to want to stick to the sphinx-based stack;

Yes, this is true -- and something that we are very mindful of. In the long run, it might be possible that this won't hold true; the papyri project is working towards documentation IR generation that builds MyST AST as an output. But, baby steps!

[choldgraf]

for one thing, some people like developers, who used autodoc and similar features, are going to want to stick to the sphinx-based stack

To make this clear: in the future, if you're running Jupyter Book >= 2.0, then Sphinx will no longer be available to you. MyST is an entirely different back-end with a different configuration structure. If you want to continue using Sphinx via Jupyter Book you'll need to pin to < 2.0 - the Sphinx extensions it uses may be maintained to some degree.

As @agoose77 notes - there might be ways to incorporate more developer-focused workflows into the MyST engine (e.g. via papyri), though its initial focus will be on computational narratives rather than software documentation.

[parmentelat]

right; so to make things clear, I am myself in the teachers category, and I totally agree that sphinx is not the right tool for me
so I'm all for the move; I was just taken a bit aback by this notion that one day the jupyter-book command would behave in a totally different way;
but I think I get the struggle you're in in terms of naming/branding - particularly as the myst/mystjs/mystmd choice of names has been particularly unfortunate for where you're headed, imho, if I may say :-)

anyways, thanks for the discussion and all the insights, this is helpful :)

[choldgraf]

Could you give more information on what you think is problematic about the name Myst? That would be helpful to think about

[parmentelat]

oh; imho jupyter book hints at publishing; markdown myst refers to a markup language, that people can easily assimilate to quick and not-too-powerful replacement for html - think whats'app or github or social media..

so imho the second hints at "you can do simple stuff quickly but don't demand too much"
it's as if you were removing the publishing dimension... which is exactly not what you're doing...

(it took me more than a year to consider that mystjs, that in the meanwhile had become mystmd, could maybe indeed be a modern publishing system ;)

[choldgraf]

That's a really helpful insight. Do you have thoughts on how we can balance the "brands" of the two projects? I think we do want to keep the jupyter book brand clear, and focus it around the "multi document books and knowledge bases" usecases of Myst. Then myst can be a more flexible document engine (like sphinx) and jupyter book builds on top of it for a specific use case .

[parmentelat]

mmh, well, I'm clearly reaching the borders of my comfort zone here :)
with that said, here are a few candid thoughts about that matter, FWIW:

I understand the need to leverage on the 'jupyter book' success story so far
however, the initial phrasing by @agoose77 that

jupyter book is just an alias for myst

triggered my reaction because, as far as I can tell right now - and please show me wrong :) - there is going to be, at least in the short-to-mid term, substantial differences in both tools capabilities; particularly with respect to using various sphinx extensions IIUC; I have not a clear view of the boundaries here, but it seems to me that the differences are big enough (and may remain so over time) that you need IMHO to propose an opt-in mechanism, if only to preserve a minimal amount of backwards compatibility

so here's a few brain dump ideas of how you could consider "rebranding" the new stack
jupyter mystbook (a little long admittedly)
jupyter bookmyst
jupyter book2 (a little awkward, but shortens as jb2)
jupyter mbook
jupyter webbook
...

my $0.02, obviously..

regardless, this is all very exciting, so thanks a lot again for all the work and the awesome tools, I can't wait to see where this all goes :)
(and thanks also to @agoose77 for pointing to jupyter/papyri, was not aware of that one either..)

[choldgraf]

Thanks for this @parmentelat ! Would it be acceptable to recommend people just use pip install jupyter-book<2 if they wish to keep using the Sphinx stack? This should work to install the Sphinx version of Jupyter Book, and we can keep the documentation at something like jupyterbook.org/v1.

[parmentelat]

I take it you mean jupyter-book<2

well from a purely selfish perspective, it'd be kind of workable, given that I'm the only one who'd need to mess with that (as opposed to, having to instruct my students, that'd be another business entirely ;-)

but from a more general standpoint, I could understand if some people were to find that a bit harsh...
at the very least you'll want to be nice and warn users well in advance, a bit like when jlab4 "absorbed" nbclassic, with a very visible message :)

[choldgraf]

Totally agree - I think it is critical both to help people downgrade if they want w helpful messages, and to upgrade as easily as possible.

What would be your wishlist of functionality you'd want to see with the myst engine that seems to be only possible with Sphinx right now?

[parmentelat]

ok, since you're asking, here are the big showstppers that I see at this point

• support for the new toc thing, with content-less labels and fine-grained control on what is collapsible or not (the initial topic of this thread in fact :), so that I can see if/how I can improve my TOCs that are currently a little awkward because of the sphinx data model
• also in my case, hosting under readthedocs is crucial for me to deploy
  • ReadTheDocs hosting support #1133
• the abridged URLs won't play well at all for me
  • Support folder hierarchies and nested files in URLs without flattening them #670

I have also tried to make a tour of the features that I need - not an easy task - and could probably give more details on a few things that stick out; at first glance:

• looks like HTML() and IFrame() from the IPython.display module don't do what they used to (see next comment)
  • Jupytext notebook support #1245
• I am also struggling with relative URLs; there was some magic linked to storing stuff under _static, and I take it this was another sphinx thing

so well, I guess I'm going to have to wait a little longer before I can dive and try to migrate a sample course book for real :-)

[parmentelat]

but wait, it feels like I'm missing something big here; I am only now realizing that my notebooks - which are jupytext-saved - are being treated as text, and simply not being run at all... which accounts for the note on HTML() above, please ignore that one

[choldgraf]

@parmentelat thanks for this feedback - I've taken the liberty of editing your comment to link the respective issues tracking each of these things (and opened a new one for Jupytext #1245, could you provide any additional information in that issue?). I hope that was OK!

I am also struggling with relative URLs; there was some magic linked to storing stuff under _static, and I take it this was another sphinx thing

Could you go into more detail on this? I'm not sure I understand.

[agoose77]

This issue itself can be closed :)