nixos/xserver: Implement configuration of NVIDIA Optimus via PRIME

[ambrop72]

This adds configuration options which automate the configuration of NVIDIA Optimus using PRIME. This allows using the NVIDIA proprietary driver on Optimus laptops, in order to render using the NVIDIA GPU while outputting to displays connected only to the integrated Intel GPU. It also adds an option for enabling kernel modesetting for the NVIDIA driver (via a kernel command line flag); this is particularly useful together with Optimus/PRIME because it fixes tearing on PRIME-connected screens.

The user still needs to enable the Optimus/PRIME feature and specify the bus IDs of the Intel and NVIDIA GPUs, but this is still much easier for users and more reliable. The implementation handles both the X configuration file as well as getting display managers to run certain necessary xrandr commands just after X has started.

Configuration of commands run after X startup is done using a new configuration option services.xserver.displayManager.setupCommands. Support for this option is implemented for LightDM, GDM and SDDM; all of these have been tested with this feature including logging into a Plasma session.

Note: support of setupCommands for GDM is implemented by making GDM run the session executable via a wrapper; the wrapper will run the setupCommands before execing. This seemed like the simplest and most reliable approach, and solves running these commands both for GDM's X server and user X servers (GDM starts separate X servers for itself and user sessions). An alternative approach would be with autostart files but that seems harder to set up and less reliable.

Note that some simple features for X configuration file generation (in xserver.nix) are added which are used in the implementation:

• services.xserver.extraConfig: Allows adding arbitrary new sections. This is used to add the Device section for the Intel GPU.
• deviceSection and screenSection within services.xserver.drivers. This allows the nvidia configuration module to add additional contents into the Device and Screen sections of the "nvidia" driver, and not into such sections for other drivers that may be enabled.

Motivation for this change

Setup of Optimus laptops is extremely hard and error-prone and in grave need of automation. Many users probably just give up even though Optimus laptops are technically supported by the NVIDIA driver via this setup.

It would be good for others to do some extra testing before merging this:

• On Optimus laptops.
• On non-Optimus computers with NVIDIA cards (when this feature is off, I cannot test this).

P.S. I am aware of Bumblebee but I never bothered with it. Do we need checks to prevent PRIME and Bumblebee being enabled at the same time?

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Fits CONTRIBUTING.md.

---

[ambrop72]

I made a fix in the definition of the setupCommands option, the default needs to be "" not [].

[dermetfan]

Tried it, works for me! At last I see the nvidia card in glxinfo output. Thanks!

[lheckemann]

👍 on making this work. Haven't tested it though.
👎 for merging the gdm patches into one. It's good to have the changes justified separately and individually.

[ambrop72]

I have updated it to have separate patches in GDM.

[dukzcry]

So this makes nvidia card always on? The idea of optimus was to enable nvidia card only on need e.g. for running graphics intensive program and keep it off in other time to save laptop battery. Why not just use one of these two solutions? https://nixos.wiki/wiki/Nvidia#Optimus

[ambrop72]

@dukzcry Yes, this makes the nvidia card always on and used for all rendering. As far as I know, this is the only approach to use the NVidia driver (and get good 3D performance) on mux-less Optimus laptops which is officially supported by NVidia. The approaches you linked to are not supported by NVidia and sometimes work, sometimes don't or are partially broken in more-or-less subtle ways. Note that many people do not mind the increased power usage for the benefit of things just working with simple configuration.

[dukzcry]

@ambrop72

Yes, this makes the nvidia card always on and used for all rendering

Then you should make a warning about it in documentation, because this is not a way it meant to be used.

The approaches you linked to are not supported by NVidia and sometimes work, sometimes don't or are partially broken in more-or-less subtle ways

The second solution is exactly official nvidia prime solution http://us.download.nvidia.com/XFree86/Linux-x86/358.16/README/randr14.html

Note that many people do not mind the increased power usage for the benefit of things just working with simple configuration

Nvidia optimus is not properly brought to linux by nvidia. So there is no proper nix way to use it. Only non-nix ways, until vendor will do something. There is a slow progress though:
#37369
https://www.phoronix.com/scan.php?page=news_item&px=Server-Side-GLVND
https://www.phoronix.com/scan.php?page=news_item&px=X.Org-Server-1.20-Feature-List

[ambrop72]

@dukzcry The NVidia page you link to is what this pull request enables - doing all rendering on the NVidia GPU but allowing output to display connected only to the Intel GPU. See that the X configuration done by my changes is that that page lists under "Older X servers require...". (Note: I did that not the simpler configuration because that does not work on certain hardware like on my laptop.)

The second solution in https://nixos.wiki/wiki/Nvidia#Optimus ("Nvidia PRIME") seems to be an unofficial approach for using the NVidia GPU only for specific applications (note that the nvidia page does not mention primerun).

I will update the documentation to mention that the NVidia GPU will be always on.

[ambrop72]

@dukzcry Also that wiki page does not mention that primerun starts a new X server for a single application which then cannot integrate with the running desktop. This certainly feels like something that users need to be aware of when considering solutions.

[rimmington]

NVIDIA => Intel?

[ambrop72]

I have updated descriptions:

• First paragraph for hardware.nvidia.optimus_prime.enable improved to explain the NVIDIA GPU will be always on, and that this enables usage of mux-less outputs.
• NVIDIA -> Intel in two places in hardware.nvidia.optimus_prime.intelBusId.

[dukzcry]

@ambrop72

See that the X configuration done by my changes is that that page lists under "Older X servers require...". (Note: I did that not the simpler configuration because that does not work on certain hardware like on my laptop.)

Thanks, this is new info to me. Because they put that simplification was possible due X server update, and no word that simple config leads to issues on some hardware.

The second solution in https://nixos.wiki/wiki/Nvidia#Optimus ("Nvidia PRIME") seems to be an unofficial approach for using the NVidia GPU only for specific applications (note that the nvidia page does not mention primerun).

Well both nvidia-xrun (not supported on NixOS), primerun (supported on "any" distro) and your solution work as written at http://us.download.nvidia.com/XFree86/Linux-x86/358.16/README/randr14.html The key difference is be it main X server or secondary temporary X server.

Also that wiki page does not mention that primerun starts a new X server for a single application which then cannot integrate with the running desktop. This certainly feels like something that users need to be aware of when considering solutions.

You're right. It doesn't matter in case of game programs though.

[dukzcry]

We already have hardware.nvidiaOptimus (currently only to disable optimus card). Perhaps you could combine namespaces into something like hardware.nvidia.optimus?

[lheckemann]

bumblebee might be worth integrating there too.

[ambrop72]

I'm not sure about this, I put the three options into the hardware.nvidia.optimus_prime section because one enables the Optimus/PRIME feature and the other two define required parameters for using it (bus IDs). I wouldn't want to have this and the existing options like hardware.nvidiaOptimus.disable (which I not familiar with) and hardware.bumblebee all thrown in the same place. Maybe these two could be moved to somewhere under hardware.nvidia since they are both NVidia related? Possibility: hardware.nvidia.disableOptimus, hardware.nvidia.bumblebee (also: Bumblebee is not a piece of hardware).

[ambrop72]

Though we might indeed have an optimus subsection, ending up with:

• hardware.nvidia.optimus.prime
• hardware.nvidia.optimus.bumblebee
• hardware.nvidia.optimus.disable (but maybe there is a better name for this)

[matthewbauer]

👍 for hardware.nvidia.optimus.prime. And then eventually move hardware.nvidiaOptimus.disable to hardware.nvidia.optimus.disable, and hardware.bumblebee to hardware.nvidia.bumblebee. We can wait on the latter part though.

[matthewbauer]

Looking at https://wiki.archlinux.org/index.php/NVIDIA_Optimus it appears there is another option where you use the proprietary NVIDIA drivers.

[ambrop72]

@dukzcry I found the hint do to the "old style" config here after searching for the "No devices found" error: https://devtalk.nvidia.com/default/topic/1027345/linux/why-cannot-i-enable-drm-kernel-mode-setting-/post/5225539/#5225539 - it seems to be an issue that appears only when you enable the modesetting. I have no idea why this isn't mentioned in any of the official places.

[rimmington]

I've cherry-picked this on top of 18.03 (specifically 91b286c) and can finally get the HDMI out on my HP Pavilion 15-cb063tx to work properly. Thanks!

[ambrop72]

Hey, is this going to be merged?
I'm not sure if there is consensus regarding the naming. I can rename these options to hardware.nvidia.optimus.prime if we agree that is a good idea. Otherwise I think this should be merged.

[matthewbauer]

@GrahamcOfBorg build gdm

[GrahamcOfBorg]

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: gdm

Partial log (click to expand)

Cannot nix-instantiate `gdm' because:
error: attribute 'gdm' in selection path 'gdm' not found

[GrahamcOfBorg]

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: gdm

Partial log (click to expand)

Cannot nix-instantiate `gdm' because:
error: attribute 'gdm' in selection path 'gdm' not found

[GrahamcOfBorg]

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: gdm

Partial log (click to expand)

Cannot nix-instantiate `gdm' because:
�[31;1merror:�[0m attribute 'gdm' in selection path 'gdm' not found

[matthewbauer]

@GrahamcOfBorg build gnome3.gdm

[GrahamcOfBorg]

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: gnome3.gdm

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

[GrahamcOfBorg]

Success on x86_64-linux (full log)

Attempted: gnome3.gdm

Partial log (click to expand)

shrinking /nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2/bin/gdmflexiserver
strip is /nix/store/1hi76hr87bd1y1q1qjk0lv8nmcjip1c8-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2/lib  /nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2/libexec  /nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2/bin  /nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2/sbin
patching script interpreter paths in /nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2
/nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2/etc/gdm/PreSession/Default: interpreter directive changed from "/bin/sh" to "/nix/store/dkh7l9a4sx7zqh8riqbj3z21sz25p8xy-bash-4.4-p23/bin/sh"
/nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2/etc/gdm/Init/Default: interpreter directive changed from "/bin/sh" to "/nix/store/dkh7l9a4sx7zqh8riqbj3z21sz25p8xy-bash-4.4-p23/bin/sh"
/nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2/etc/gdm/PostSession/Default: interpreter directive changed from "/bin/sh" to "/nix/store/dkh7l9a4sx7zqh8riqbj3z21sz25p8xy-bash-4.4-p23/bin/sh"
/nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2/etc/gdm/PostLogin/Default.sample: interpreter directive changed from "/bin/sh" to "/nix/store/dkh7l9a4sx7zqh8riqbj3z21sz25p8xy-bash-4.4-p23/bin/sh"
checking for references to /build in /nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2...
moving /nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2/sbin/* to /nix/store/z2rj5kvp7sagfmaz2vw555vpn9vljk7i-gdm-3.28.2/bin

[GrahamcOfBorg]

Success on aarch64-linux (full log)

Attempted: gnome3.gdm

Partial log (click to expand)

strip is /nix/store/zrs21zqcchgyabjf4xfimncdq16njizc-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2/lib  /nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2/libexec  /nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2/bin  /nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2/sbin
patching script interpreter paths in /nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2
/nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2/etc/gdm/PostSession/Default: interpreter directive changed from "/bin/sh" to "/nix/store/6vqgi9d49smsbr2qxra52yhipg0yxf9f-bash-4.4-p23/bin/sh"
/nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2/etc/gdm/PreSession/Default: interpreter directive changed from "/bin/sh" to "/nix/store/6vqgi9d49smsbr2qxra52yhipg0yxf9f-bash-4.4-p23/bin/sh"
/nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2/etc/gdm/PostLogin/Default.sample: interpreter directive changed from "/bin/sh" to "/nix/store/6vqgi9d49smsbr2qxra52yhipg0yxf9f-bash-4.4-p23/bin/sh"
/nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2/etc/gdm/Init/Default: interpreter directive changed from "/bin/sh" to "/nix/store/6vqgi9d49smsbr2qxra52yhipg0yxf9f-bash-4.4-p23/bin/sh"
checking for references to /build in /nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2...
moving /nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2/sbin/* to /nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2/bin
/nix/store/nfk1b67ybdqc80xbc3jaihhi4ib5cqq7-gdm-3.28.2

[vcunat]

/cc maintainer @jtojnar about the gdm patches (though they appear good at a superficial look by me).

[jtojnar]

Will check it later, will need to be checked for compatibility with #43992 @hedning

[hedning]

@ambrop72 GDM have a few script entry points that can probably be used. In particular /etc/gdm/Init/Default and /etc/gdm/PreSession/Default should probably serve your purpose (not entirely sure if these scripts can modify the environment, but that doesn't seem to be needed here anyways). If that's workable it would be far better than patching GDM I think.

With regards to #43992 this doesn't really seem to conflict in any meaningful way I think, might not apply cleanly (doesn't seem to confilct), but the functionality seems distinct so should probably be easy to merge.

[ambrop72]

@hedning The /etc/gdm scripts no longer actually work despite being in the docs. See https://bugzilla.redhat.com/show_bug.cgi?id=449675

[hedning]

Hmm, I'm able to run the PreSession script at least, but yeah, Init doesn't work (looks like the code responsible is never called) so I'm guessing that's not usable.

[ambrop72]

I just tested this again with recent master, it merges without problems and works fine with all three display managers (sddm, lightdm, gdm).

[ambrop72]

Why is this not being merged? I think it would be very good to include this in 18.09, many people will finally get their graphics working well.

[mtrsk]

Is there a way to test this even if it's not merged? I've tried the first wiki's solution (with both 18.03 and unstable) and some other configs found in stackexchange and they didn't seem to work.

I'm new to NixOS btw.

[dermetfan]

@mschonfinkel

1. clone the nixpkgs repo
2. add ambrop72/nixpkgs as remote
3. check out the optimus-prime-config-master branch
4. enable options from this PR in your configuration.nix
5. nixos-rebuild switch -I nixpkgs=/path/to/your/clone

[ambrop72]

@mschonfinkel, @dermetfan
Doing that literally is not a good idea because the pull request hasn't been updated in some time and is not based on a tested channels release. Instead one should check out the latest 18.09 channel from the nixpkgs-channels repository and then manually merge this:

git clone --single-branch -b nixos-18.09 https://github.com/NixOS/nixpkgs-channels.git nixpkgs
cd nixpkgs
git remote add ambrop72 https://github.com/ambrop72/nixpkgs.git
git fetch ambrop72 optimus-prime-config-master
git merge ambrop72/optimus-prime-config-master

And then configure and build like @dermetfan wrote.

[mtrsk]

It's working now 💯 ! Thanks @ambrop72 and @dermetfan!

[leotaku]

Could we maybe get this merged before 19.03? Consistent Nvidia support seems like a great improvement and also advertisement for a mayor release.

[coretemp]

It's not a good thing when multiple people need to beg to get something merged. Is anything being done to address this? Looking at the history it should have been merged 3 months ago.

In particular, it's frustrating for the opener of the PR. It sends the message that contributions are not welcome.

[matthewbauer]

I definitely agree this has been in the works a little too long. However, at some point we need to resolve the conflict between the naming schemes of all the nvidia stuff. That's been my main hesitancy so far. We can look into backporting this to 18.09 once that has been resolved. Besides naming though it looks to be a good addition.

[worldofpeace]

@matthewbauer

Can we outline what changes need to be made to get this into 18.09?

I went through the process today of explaining to someone who has never used nixos before, in IRC, how to setup a local nixpkgs checkout for this among other things. This module was very helpful.

[ambrop72]

For 18.09, merging/cherry-picking the commit in this pull request will work. I confirm it works with SDDM and LightDM.

However, it does not work with GDM but that seems to be an unrelated "GDM no longer works with NVidia" problem. The fatal error is NVIDIA(GPU-0): Failed to initialize the NVIDIA GPU at PCI:1:0:0. ... and earlier there is xf86EnableIOPorts: failed to set IOPL for I/O (Operation not permitted). Possibly caused by not running X as root?

[ambrop72]

About the GDM problem, the issue is indeed related to X running as non-root but it is specific to this setup not NVidia in general. The problem is that for this to work, X needs permission to access the VGA arbiter (/dev/vga_arbiter). I don't know what the proper solution here would be, but giving world access to this is a workaround that gets GDM to work:

  services.udev.extraRules = ''
    KERNEL=="vga_arbiter", OWNER="root", GROUP="root", MODE="0666"
  '';

[matthewbauer]

I went ahead and cherry-picked it in 195a573 for now. I was just hoping we could get consensus on what the "right" name for all of these Optimus/Bumblebee/Prime/etc. names was. Hopefully we can clean it up later, though. It's more bikeshedding than anything.