host Nix reference manual on nix.dev

[fricklerhandwerk]

this is the most cursed setup you will see any time soon.

we're dumping the Nix manual unchanged into the build tree after
building. the reason is that we'd want to link to it from our table of
contents, but because Sphinx does not allow adding arbitrary files to
the build output in arbitrary locations (only _static works). but we
want to place the manual behind a particular URL, and the alternative of
maintaining URL rewrites (e.g. in nginx) is not accessible here because the
infrastructure is managed somewhere else.

now that the files won't appear in their desired locations at Sphinx
build time, we can't use relative links to reference them, therefore we
have to resort to pointing to the web URL the manual will appear at.
this is terrible and I'm sorry. please fix this if you have a better
idea. once we use URLs though, we have to avoid linkchecking, since
those files may not be there before deploying them.

figuring all of this out took way longer than anyone would wish.

thanks @alejandrosame for enduring the process.

[github-actions]

Deploy Preview

Name	Result
Last commit:	597acd72f57ed398bf4219c57a249a664d6afecd
Preview URL:	https://d3c2049e.nix-dot-dev.pages.dev

[zmitchell]

• Why are we using the manual from 2.13 and not the current manual?
• The sidebar link to the manual is broken

[fricklerhandwerk]

Why are we using the manual from 2.13 and not the current manual?

Because that's the version used in the latest NixOS release, while the most recent one will contain all kinds on things that don't exist there yet.

In the next steps we should figure out how to make other versions easily discoverable once they're available. One way would be to list them in the glossary where the manuals are also linked.

The link to the manual is broken only in the preview deployment, for reasons explained in the PR description.

[zmitchell]

It definitely feels weird to me to have the Nix manual linked from nix.dev to be out of date compared to the one that you'll find by searching elsewhere. That's also likely to surprise users.

[fricklerhandwerk]

Okay, so these are two different concerns. We can flip the version as we wish now that we have the infrastructure, that's cheap.

The more interesting question is how to present the fact that NixOS uses a particular version of Nix and that you'll likely want to use the manual for that rather than some random version from the future. One thing I considered was to actually pin the installation instructions to the same version as well. Because in the user journey we're building up, what may end up happening is that you install the latest version of Nix on Ubuntu, then add Home Manager which pins Nix to 2.13, and suddenly you can't run builds any more because the store DB schema is not forward compatible.

This is something I could already add in this PR to make the transition smooth for people starting out here. We can't do it in perfect lockstep, but the accompanying change will be to remove installation instructions and the manuals from nixos.org. The blocker here is though that I'd first want to prepare the Nixpkgs and NixOS manual upstream builds to use the right styling, as otherwise we'd have to process them here again. This is something I'm also currently working on, but it will take a bit of time, maybe another week or two on the wall-clock depending on distractions, as I have multiple ongoing commitments.

@zmitchell would you suggest to block the incremental change on everything else being finished, to avoid inconsistency? I'd prefer a temporary inconsistency to get it off my stack.

[zmitchell]

So you could do all of that, or you could just have two links:

• Nix Manual (Current)
• Nix Manual (NixOS 23.05)

That seems like a better solution

[fricklerhandwerk]

Okay, added a separate page where we can tall all we want about the Nix manual. @yukiisbored is there a way to force-expand only one entry of the TOC in the sidebar? Then users wouldn't have to click through to the page.

But I think this is useful now. @thufschmitt please take a look.

[github-actions]

Deploy Preview

Name	Result
Last commit:	3fea7ff960d8f229c726313de6c570a12cb040d5
Preview URL:	https://6f63204d.nix-dot-dev.pages.dev

[thufschmitt]

Thanks, @fricklerhandwerk !

What's the plan from the nixos.org side? Will we keep the manual on both websites or do we have a path towards redirecting the manual from nixos.org to here?

[fricklerhandwerk]

Once this is merged I'll migrate the per-page redirects here, and then put in place a manual-wide redirect on nixos.org.

Eventually the page redirects should move upstream into the Nix manual, but that needs a bit of figuring things out.

[asymmetric]

@fricklerhandwerk is it known that the links to the manual for 2.13 and 2.18 are broken?

[fricklerhandwerk]

@fricklerhandwerk is it known that the links to the manual for 2.13 and 2.18 are broken?

The public URLs cannot be populated before deploying this PR. See the commit message for why it's done this way. The deploy preview hosts them under the relative path: https://6f63204d.nix-dot-dev.pages.dev/manual/nix/2.13/

[yukiisbored]

@yukiisbored is there a way to force-expand only one entry of the TOC in the sidebar?

I don't think that is possible to do with Sphinx TOC.

Though, mdbook has the version widget maybe we should look into using that instead.

[fricklerhandwerk]

That version widget is slapped on top by nixos.org. I'd prefer to avoid porting that and do more things statically. That's easier to find, to understand and to maintain. (Documentation should be low-tech!)

[infinisil]

This feels too hacky to me :/

[fricklerhandwerk]

@infinisil @zmitchell yes, this is hacky and I'm open to suggestions. Opened NixOS/nix#9275 as a pathway to fix forward.

@thufschmitt added another Nix release.

[github-actions]

Deploy Preview

Name	Result
Last commit:	6c18a10211980711818725a27277bc04dfb9822c
Preview URL:	https://2dae7df6.nix-dot-dev.pages.dev

[infinisil]

We should document everything that needs to be done regularly:

• Updating Flake inputs
• Adding new Flake inputs
• Doing string replacement on the URLs

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-11-02-documentation-team-meeting-notes-91/34938/1

[infinisil]

Needs a rebase, but otherwise this looks okay to me.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-11-27-nix-team-meeting-minutes-107/36112/1

[roberth]

Thoughts:

• Add redirects for /manual/nix/stable and the nixos releases? We'd first have to decide what the names should even be, so let's postpone.
• Maybe get the manuals from Nixpkgs if applicable? Not sure, because we can't do that for all versions we want to show, and we'd have to pick flake vs Nixpkgs arbitrarily for most versions.
• Commit the generated sites, so that the build process and particularly the evaluation, are more scalable? Not essential and has its own complexity just like the current PR approach. Let's try this PR first.

100% love the concept. nix.dev isn't nixos.org and Nix isn't NixOS, so that gets rid of what was arguably a misnomer. Also explicit versions are good. We might want to keep more versions around, but that's not a matter of urgency, as mentioned.

Code looks ok. Let's see how this goes in practice.

[edolstra]

This is really confusing because now we have two canonical locations for the Nix manual, since https://nixos.org/learn points at the Nix manual at https://nixos.org/manual/nix/stable/.

nix.dev isn't nixos.org and Nix isn't NixOS

nixos.org is the official homepage for Nix. Changing that should be a decision for the marketing team.

Also, this PR makes me have little faith in the ability of nix.dev to represent the direction of the Nix project.

[fricklerhandwerk]

now we have two canonical locations for the Nix manual, since https://nixos.org/learn points at the Nix manual at https://nixos.org/manual/nix/stable/.

This is very much temporary, and can't be switched over atomically anyway, because the moving parts live in different repositories. It will be fixed by adding a redirect from nixos.org after merging #829.

[06kellyjac]

On the marketing team comment I can agree but for all I know there might have been discussions with the marketing team already.

---

Also, this PR ...

I like using flakes and I personally would have just kept using them, but it is objectively true that there are users that don't use experimental features and or aren't happy with flakes yet.

The nix project "represents" flakes as "experimental" so it is completely fair for a project representing nix to show the other options currently in use.

Even if the experimental flag was removed are you suggesting every single project under the nixos org umbrella must use them?

The impression I've got from the recent calls to remove the experimental tag is "well flakes are in a good spot, we can continue improving them when non-experimental, you can use flakes or not just like you can now and its totally a non issue" but this kind of goes against that important last part.

It only makes sense to enforce the usage of flakes for the nixos org projects if the non-flakes approach are explicitly viewed as legacy or deprecated.

[asymmetric]

the reason is that we'd want to link to it from our table of
contents, but because Sphinx does not allow adding arbitrary files to
the build output in arbitrary locations (only _static works). but we
want to place the manual behind a particular URL, and the alternative of
maintaining URL rewrites (e.g. in nginx) is not accessible here because the
infrastructure is managed somewhere else.

@fricklerhandwerk there is a way actually:

• placing the rendered 2.19 manual in e.g. ./source/_static/manuals/2.19
• linking to it with bare <a> elements (because sphinx/myst don't know about it, so stuff like this doesn't work
• adding a bunch of symlinks like ./source/_static/manuals/latest -> 2.19

This works (at least locally), with a few downsides:

• the rendered maual is big (32MiB, which can be reduced to ~22 by removing a few files)
• because of this, the git repo size would go up
• we need to set up a workflow that does the manual size reduction
• the sidebar links to the manuals would need to remain hardcoded to https://nix.dev (due to limitations with the toctree directive)

Advantages:

• we can use relative links
• less Nix magic to build the wbsite???

[fricklerhandwerk]

That sounds like an even worse hack. 😐