nixpkgs manual in CommonMark using sphinx+myst and jupyter-book

[FRidh]

Motivation for this change

If we change to CommonMark, I prefer if we go for MyST, as that allows one to fall back to reStructuredText. MyST also comes with plenty of extensions, including internal references which I think are critical.

This PR uses jupyter-book which has quite a lot of dependencies. We could also fall back to just Sphinx + MyST, it does not matter much really I think. All in all it seems jupyter-book is a nicer ready solution.
https://jupyterbook.org

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

TODO

• fix tests of new python packages
• sphinx setting: elevate warnings to errors (not yet done because pygments somehow doesn't recognize nix markup)
• create a toc
• refer to all existing md docs
• convert remaining docs from docbook to md in a nix build
• include manual in release*.nix
• include functions reference
• send sphinx update through staging :(
• logo

For the time being we would have two builds, but with basically the same contents.

$ nix-build doc-md

[ryantm]

What is jupyter-book? Is it really close enough to CommonMark to be in the spirit of RFC 72?

[FRidh]

https://myst-parser.readthedocs.io/en/latest/index.html

Yes, language wise it is just MyST which is a superset of CommonMark https://myst-parser.readthedocs.io/en/latest/index.html.

[ryantm]

I don't know much about the various options available, but I have a preference for matching whatever Nix is already doing if it is acceptable https://github.com/NixOS/nix/tree/master/doc/manual

[jonringer]

if you bump sphinx, I think this will have to go in staging

[FRidh]

As far as I can tell everything is now functioning.

We now need to test the impact of the Sphinx upgrade and then send that to staging. Other packages can also be merged after tests are ok.

[jtojnar]

One disadvantage of MyST is that the syntax is even more convoluted than XML – we would replace English-like tags with weird and unmemorable combinations of symbols. Using CommonMark with MDX has much more uniform syntax.

[stale]

I marked this as stale due to inactivity. → More info

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/any-consensus-on-documentation-generation-tool-for-nixpkgs-manual/15550/5

[SuperSandro2000]

There are a lot of new lines missing to make git happy.

Please complete meta everywhere with maintainer and homepage.

All packages should have a pythonImportsCheck.

Please quote all homepage urls.

Can we fetch from GitHub to enable tests for some of the packages?

[SuperSandro2000]

This should be after version.

[SuperSandro2000]

Why are tests disabled?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-01-19-documentation-team-meeting-notes-24/25537/1

[SuperSandro2000]

The docs rework is done and this PR has accumulated quite a lot of merge conflicts in the meantime. If we want to keep some of the packages, I think we would need to dust them off first and should probably cherry-pick them into another PR.

Closing because of the above mentioned things.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-01-25-documentation-team-meeting-notes-106/38792/1