treewide: markdown option docs

[pennae]

Description of changes

just an attempt at getting option descriptions to markdown (cf #175586).

the conversion is obviously not completely faithful (since representing the difference between <filename> and <literal> isn't easy to represent in markdown, let alone very useful?), but minimizing the diff between the current options.xml and the one after md conversions is an explicit goal. once all descriptions have been converted we can drop mdDoc again in a mechanical patch.

on top of the syntax extensions the manual already uses this introduces two new ones: {command}`foo` and {option}`foo`. both render to their docbook <tag>foo</tag> equivalent to keep the conversion between docbook descriptions and md as faithful as possible. in the future we may want to use {option} to auto-link option uses to their descriptions, but right now {command} and {option} have no such special functions.

one problem we see with these extensions is that the processor used for the options manual would have to support them, which at least for mddoc seems very hard to achieve at first glance. keeping {option} doesn't seem that important, but {command} is rendered differently from regular literals in the manpage build (so we should probably keep that, somehow). keeping the admonitions also seems very useful, not least because they stand out nicely even in the manpage build.

cc @roberth

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[roberth]

This fits all the requirements expressed in #175586 at this point, with the exception of @edolstra's dislike of mdDoc, although that was in conjunction with pandoc.

I suppose another potential complaint is that we're using a markdown processor, mistune, that isn't the one used for the rest of the manual, and isn't the one used in the Nix (package) manual either. It is "compatible with sane CommonMark rules."
We may want to review the plugins but otherwise it seems we're good.
If someone can find a good reason to switch to a different markdown processor, they are free to do so during or after this PR, but that should not block migration progress.

The implementation looks ok so far. Will give this a try later.

Looks like we can finally complete the markdown migration! 🎉

---

the difference between <filename> and <literal> isn't easy to represent in markdown

tl;dr not a problem.
I am not aware of anything that uses this information. The only potential use I can think of is producing links somehow, but it doesn't really hold enough information to produce such links. If we want links, we should just add links.

[roberth]

Diffing seems like a good idea.

Instead of trying to match every quirk, we can choose to canonicalize both inputs to the diff. We'll need something like that for <filename> anyway.

[pennae]

I suppose another potential complaint is that we're using a markdown processor, mistune, that isn't the one used for the rest of the manual, and isn't the one used in the Nix (package) manual either. It is "compatible with sane CommonMark rules."
We may want to review the plugins but otherwise it seems we're good.

we really only chose mistune because it's a commonmark parser available in python that doesn't enlarge the docs closure by too much. anything else that can parse markdown into an ast would be fine. even shelling out to the favored markdown processor and have it dump an ast wouldn't be a problem, we think.

[pennae]

• added literalMD for examples and defaults
• fixed link handling and escaping
• keep language attrs on programlisting (no option uses this so far)
• added lists and emphasis

things we're not sure how to convert:

• <option>? probably best to turn them into local links
• <replaceable>? would suggest <replaceable>X</replaceable> → «X» since this matches how attrset placeholders are rendered
• <note>/<warning>/etc? nixos markdown has admonitions that would be useful here, maybe mistune can be made to support them with plugins/parser rules. haven't checked that yet

[pennae]

admonitions are indeed possible :)

[pennae]

updated description to the current state of the patch set. apart from a few kinds of admonitions the conversion script should be mostly complete (if still very experimental).

[pennae]

does this need more modules converted as examples? 🤔

[roberth]

• converted entries render as usual
• non-converted entries render as usual
• implementation looks good

does this need more modules converted as examples?

I don't think it helps with review and merge. If you're not confident that others can contribute conversion PRs, you could do some more, but leave them for a follow-up PR.

Some documentation will be helpful:

• Nixpkgs Contributing to this documentation/Syntax
  • this already explains the existing markdown syntax used in manuals. Is it complete for the purpose of options? Any differences?
• NixOS Writing Documentation
  • reference the Nixpkgs manual?
• N/A? Temporary-ish process documentation will suffice, e.g. github issue. NixOS Contributing to this manual
  • would be nice to have some links to the other sections

When those are done and it's ready for other to start contributing, I think we're ready for a merge.

[jtojnar]

What do you mean by trailer?

[pennae]

trailing whitespace that is not rendered. we keep trailing whitespace at the end of a description to make diffing options.xml before and after a migration easier, but that isn't possible here. apart from that there is not reason to care about whitespace and if we use a different diffing process we can also remove this whitespace handling entirely.

[pennae]

we've added some documentation to the nixos and nixpkgs manuals clarifying which extensions are available in options docs. not entirely sure which sections to link for the final nixos manual chapter though 😕

[roberth]

which sections to link for the final nixos manual chapter though

I was also thinking about documenting the transition process, but it's perhaps better to write an issue for that, similar to the other tracking issues we have, like #24288.

[roberth]

I've tried it and it worked well.

My only findings: if you get invalid xml errors in "intermediate.xml", you've forgot mdDoc.

[pennae]

I've tried it and it worked well.

🍾!

My only findings: if you get invalid xml errors in "intermediate.xml", you've forgot mdDoc.

not sure we can do anything about that, unfortunately. though what we probably could do is print a warning (error?) if a level of options is not exclusive mdDoc or exclusively docbook. not sure how useful that would be in the face of option merging though

[roberth]

if a level of options is not exclusive mdDoc or exclusively docbook

I'm not sure if it's worth the effort. Option merging is quite rare in NixOS, so I wouldn't expect much trouble from false positives if the error message mentions the possibility. I think it's ok to mention the "bad" error message in the tracking issue. It's a temporary state of affairs anyway, during the transition.

[pennae]

invalid xml errors are probably most often caused by <⋯> links, those are easy to spot. the rest of the MD syntax should usually be valid xml, but hopefully we'll catch that in review. and as you say, temporart state—worst case, it's slightly less sightly until we complete the migration. (which hopefully happens before 22.11!)

[peterhoeg]

For reasons unknown (to me), I'm getting this:

error: builder for '/nix/store/ajymzc86n3gqanmcj2kiwqra49r825g5-lazy-options.json.drv' failed with exit code 1;                                                                                                  last 10 log lines:                                                                                                                                                                                        >                                                                                                                                                                                                  
       >           252|         type = nullOr str; 
       >           253|         description = mdDoc ''
       >              |         ^
       >           254|           Address to listen on. Listen on `0.0.0.0`/`::`
       > Cacheable portion of option doc build failed.   
       > Usually this means that an option attribute that ends up in documentation (eg `default` or `description`) depends on the restricted module arguments `config` or `pkgs`.

If I drop all the mdDoc references from nixos/modules/services/networking/mosquitto.nix, I can use nixos-rebuild again.

[pennae]

are you backporting newer modules to older revisions? does it work with documentation.nixos.options.splitBuild = false?

[roberth]

@pennae, found some improvements #179351
(not related to the previous two messages)

[peterhoeg]

are you backporting newer modules to older revisions?

This is admittedly with a custom nixpkgs, but it's building the latest unstable. The problem starting showing about a week ago.

does it work with documentation.nixos.options.splitBuild = false?

I'll try to create the smallest possible example that reproduces it and let you guys know. In the meantime I'm just patching out mdDoc.