Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/sunshine: init #294641

Merged
merged 2 commits into from
Apr 20, 2024
Merged

nixos/sunshine: init #294641

merged 2 commits into from
Apr 20, 2024

Conversation

devusb
Copy link
Contributor

@devusb devusb commented Mar 10, 2024

Description of changes

Adds a NixOS module for Sunshine, a self-hosted game stream host for Moonlight.

Closes #293298

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: documentation 8.has: changelog 8.has: module (update) This PR changes an existing module in `nixos/` labels Mar 10, 2024
@devusb devusb force-pushed the sunshine-module branch 2 times, most recently from a79a582 to 5f39bcd Compare March 10, 2024 03:49
@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels Mar 10, 2024
@so-lar-is
Copy link

It'd be great if this included an openFirewall option. Sunshine requires multiple ports, which are defined as offsets from a configurable port, to be opened. This describes which ports are necessary, here is how I did it in my implementation.

@devusb
Copy link
Contributor Author

devusb commented Mar 10, 2024

It'd be great if this included an openFirewall option. Sunshine requires multiple ports, which are defined as offsets from a configurable port, to be opened. This describes which ports are necessary, here is how I did it in my implementation.

@so-lar-is I actually just pushed that a bit ago, based on yours :) thanks.

@so-lar-is
Copy link

Mea culpa, I had the tab open for a while already :)

@devusb devusb force-pushed the sunshine-module branch 3 times, most recently from f0d3d3d to b98b834 Compare March 11, 2024 22:27
@devusb devusb mentioned this pull request Mar 11, 2024
13 tasks
@peperunas
Copy link
Contributor

Great work !

I would put the securityWrapper part - the root capabilities - under a boolean flag as sunshine would work anyway.

@Cryolitia
Copy link
Member

I would like to divide it into two module: programs.sunshine and services.sunshine. Some people(like me) may only want to start it manually form terminal unusually, instaed of a systemd service.

@devusb
Copy link
Contributor Author

devusb commented Mar 17, 2024

Great work !

I would put the securityWrapper part - the root capabilities - under a boolean flag as sunshine would work anyway.

Yeah, good idea -- technically it is required for KMS capture (see docs for reference), but making it not on by default makes someone intentionally give this service that capability.

@devusb devusb marked this pull request as ready for review March 17, 2024 20:32
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/3677

@devusb devusb force-pushed the sunshine-module branch 2 times, most recently from ad6c6d4 to 2be8cd0 Compare March 26, 2024 11:17
@Cryolitia
Copy link
Member

I would like to divide it into two module: programs.sunshine and services.sunshine. Some people(like me) may only want to start it manually form terminal unusually, instaed of a systemd service.

or programs.sunshine and programs.sunshine.autoStart like

autoStart = lib.mkEnableOption (lib.mdDoc "Clash Verge auto launch");

@devusb
Copy link
Contributor Author

devusb commented Mar 29, 2024

I would like to divide it into two module: programs.sunshine and services.sunshine. Some people(like me) may only want to start it manually form terminal unusually, instaed of a systemd service.

or programs.sunshine and programs.sunshine.autoStart like

autoStart = lib.mkEnableOption (lib.mdDoc "Clash Verge auto launch");

Fair ask -- this is not my use case, unfortunately -- not sure I'll have time to add/test it here. Could certainly be added with a later PR though.

@tymscar
Copy link
Contributor

tymscar commented Mar 30, 2024

Works great on nvidia + xorg too.

The only issue is that it doesent use nvenc. For that I use this workaround, would be great if you could add it to the service, maybe with a "useCuda" flag: #272221 (comment)

@devusb
Copy link
Contributor Author

devusb commented Mar 30, 2024

@tymscar I don't have NVIDIA anymore, but when I did it was enough to use the cudaSupport flag either globally or overridden for this specific package to enable support -- not thinking it should be necessary at the module level. Thinking you could just use the package option to do something like this

services.sunshine.package = pkgs.sunshine.override { cudaSupport = true; };

For what it's worth I don't think nvenc used to require cudaSupport, but it's been awhile since I used Sunshine with NVIDIA so not sure the current state of things.

@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 31, 2024
@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 31, 2024
@tymscar
Copy link
Contributor

tymscar commented Mar 31, 2024

Thank you!
Using the override did compile some cuda things which means it might be working, however since yesterday, the service won't see my GPU at all, no matter what I do.

Perhaps yesterday I spoke too soon and maybe I still had the manually started instance? Who knows. Maybe you can give me some pointers.

Basically it does not see the gpu, this is what the logs look like:

[2024:03:31:04:35:53]: Info: Trying encoder [nvenc]
[2024:03:31:04:35:53]: Info: Screencasting with KMS
[2024:03:31:04:35:53]: Info: /dev/dri/card0 -> simpledrm
[2024:03:31:04:35:53]: Warning: No render device name for: /dev/dri/card0
[2024:03:31:04:35:53]: Info: /dev/dri/card1 -> nvidia-drm
[2024:03:31:04:35:53]: Error: Couldn't find monitor [1]
[2024:03:31:04:35:53]: Info: /dev/dri/card0 -> simpledrm
[2024:03:31:04:35:53]: Warning: No render device name for: /dev/dri/card0
[2024:03:31:04:35:53]: Info: /dev/dri/card1 -> nvidia-drm
[2024:03:31:04:35:53]: Error: Couldn't find monitor [1]
[2024:03:31:04:35:53]: Info: Screencasting with KMS
[2024:03:31:04:35:53]: Info: /dev/dri/card0 -> simpledrm
[2024:03:31:04:35:53]: Warning: No render device name for: /dev/dri/card0
[2024:03:31:04:35:53]: Info: /dev/dri/card1 -> nvidia-drm
[2024:03:31:04:35:53]: Error: Couldn't find monitor [1]
[2024:03:31:04:35:53]: Info: /dev/dri/card0 -> simpledrm
[2024:03:31:04:35:53]: Warning: No render device name for: /dev/dri/card0
[2024:03:31:04:35:53]: Info: /dev/dri/card1 -> nvidia-drm
[2024:03:31:04:35:53]: Error: Couldn't find monitor [1]
[2024:03:31:04:35:54]: Info: Encoder [nvenc] failed
[2024:03:31:04:35:54]: Info: Trying encoder [vaapi]
[2024:03:31:04:35:54]: Info: Screencasting with KMS
[2024:03:31:04:35:54]: Info: /dev/dri/card0 -> simpledrm
[2024:03:31:04:35:54]: Warning: No render device name for: /dev/dri/card0
[2024:03:31:04:35:54]: Info: /dev/dri/card1 -> nvidia-drm
[2024:03:31:04:35:54]: Error: Couldn't find monitor [1]
[2024:03:31:04:35:54]: Info: /dev/dri/card0 -> simpledrm
[2024:03:31:04:35:54]: Warning: No render device name for: /dev/dri/card0
[2024:03:31:04:35:54]: Info: /dev/dri/card1 -> nvidia-drm
[2024:03:31:04:35:54]: Error: Couldn't find monitor [1]
[2024:03:31:04:35:54]: Info: Screencasting with KMS
[2024:03:31:04:35:54]: Info: /dev/dri/card0 -> simpledrm
[2024:03:31:04:35:54]: Warning: No render device name for: /dev/dri/card0
[2024:03:31:04:35:54]: Info: /dev/dri/card1 -> nvidia-drm
[2024:03:31:04:35:54]: Error: Couldn't find monitor [1]
[2024:03:31:04:35:54]: Info: /dev/dri/card0 -> simpledrm
[2024:03:31:04:35:54]: Warning: No render device name for: /dev/dri/card0
[2024:03:31:04:35:54]: Info: /dev/dri/card1 -> nvidia-drm
[2024:03:31:04:35:54]: Error: Couldn't find monitor [1]
[2024:03:31:04:35:54]: Info: Encoder [vaapi] failed
[2024:03:31:04:35:54]: Info: Trying encoder [software]
[2024:03:31:04:35:54]: Info: Screencasting with KMS
[2024:03:31:04:35:54]: Info: /dev/dri/card0 -> simpledrm
[2024:03:31:04:35:54]: Warning: No render device name for: /dev/dri/card0
[2024:03:31:04:35:54]: Info: /dev/dri/card1 -> nvidia-drm
[2024:03:31:04:35:54]: Error: Couldn't find monitor [1]
[2024:03:31:04:35:54]: Info: Screencasting with KMS
[2024:03:31:04:35:54]: Info: /dev/dri/card0 -> simpledrm
[2024:03:31:04:35:54]: Warning: No render device name for: /dev/dri/card0
[2024:03:31:04:35:54]: Info: /dev/dri/card1 -> nvidia-drm
[2024:03:31:04:35:54]: Error: Couldn't find monitor [1]
[2024:03:31:04:35:54]: Info: Encoder [software] failed
[2024:03:31:04:35:54]: Fatal: Unable to find display or encoder during startup.
[2024:03:31:04:35:54]: Fatal: Please ensure your manually chosen GPU and monitor are connected and powered on.

Now I know that sunshine in the config tells you how to find the GPU, and when I run those commands, I can see mine at /dev/dri/renderD128 which is the default location, so it should be fine.
Part of the logs, when it starts up, shows my monitors so that should also be fine:

[2024:03:31:04:25:05]: Info: Detecting monitors
[2024:03:31:04:25:05]: Info: Detected monitor 0: HDMI-0, connected: true
[2024:03:31:04:25:05]: Info: Detected monitor 1: DP-0, connected: true
[2024:03:31:04:25:05]: Info: Detected monitor 2: DP-1, connected: false
[2024:03:31:04:25:05]: Info: Detected monitor 3: DP-2, connected: false
[2024:03:31:04:25:05]: Info: Detected monitor 4: DP-3, connected: true
[2024:03:31:04:25:05]: Info: Detected monitor 5: DP-4, connected: false
[2024:03:31:04:25:05]: Info: Detected monitor 6: DP-5, connected: false

I wonder if perhaps this is some sort of issue with permissions? At least back when I was running sunshine myself from the command line, not as a service, it would give me similar errors when I would run it under my user, instead of root, or using sudo. Would that be something we need to add to the service or is that some configuration error on my end.

The current config I use for this service is:

services = {
  sunshine = {
    enable = true;
    package = pkgs.sunshine.override { cudaSupport = true; };
    openFirewall = true;
    capSysAdmin = true;
  };
};

I tried without capSysAdmin too, but it does not seem to work either, so that is not the issue.

Thank you again for the help!

@devusb
Copy link
Contributor Author

devusb commented Mar 31, 2024

@tymscar are you on unstable? there was a change recently that might help. does it work without this PR? if not, please open an issue — PR comments probably not the best place for this — issue will help if others have the same problem as you too.

@tymscar
Copy link
Contributor

tymscar commented Mar 31, 2024

@devusb I am on unstable. I would open an issue, but this is related to this PR rather than to sunshine overall, because if I run it myself, as root, it works fine. It's just the service here that does not.

@devusb
Copy link
Contributor Author

devusb commented Mar 31, 2024

@devusb I am on unstable. I would open an issue, but this is related to this PR rather than to sunshine overall, because if I run it myself, as root, it works fine. It's just the service here that does not.

Sorry, this is hard for me to troubleshoot because I don’t have an NVIDIA card. That said, does it work not using module running as your user but with CAP_SYS_ADMIN? That is what the module does.

@tymscar
Copy link
Contributor

tymscar commented Mar 31, 2024

Well the issue isnt nvidia specific as far as I can tell, but rather something to do with permissions. I dont think it can see my GPU at all, nvenc or not.
I mentioned in the reply earlier today that I tried with and without CAP_SYS_ADMIN and that does not help however If I would have to guess the issue might stem from this running as a user unit.

@devusb
Copy link
Contributor Author

devusb commented Mar 31, 2024

Well the issue isnt nvidia specific as far as I can tell, but rather something to do with permissions. I dont think it can see my GPU at all, nvenc or not. I mentioned in the reply earlier today that I tried with and without CAP_SYS_ADMIN and that does not help however If I would have to guess the issue might stem from this running as a user unit.

So it works for me with vaapi and AMD + Wayland (or Xorg) on unstable -- but more, asking did you try without the module (so just like you would normally run it, without this PR) as your user (so not root) but with CAP_SYS_ADMIN? Trying to see if it's something wrong with the PR, or if just running it without systemd as your user with CAP_SYS_ADMIN behaves the same way.

@tymscar
Copy link
Contributor

tymscar commented Mar 31, 2024

The way I have it working is during this commit.
Here is where I start sunshine, from my i3 config.
Here is where I define it for cuda support. (it works without this too, unlike the service in this pr, but it just does not have nvenc support)

I'm not sure how I could run that without root but with cap-sys-admin because I cant change the files in the nix store.

But its very important to note that using the service you wrote, without capsysadmin, gives me errors like Error: Failed to gain CAP_SYS_ADMIN that go away if I set it to true.

So that isnt the root of the issue here. The GPU not being detecting doesent seem to be influenced at all by it.

@devusb
Copy link
Contributor Author

devusb commented Mar 31, 2024

You can use security.wrappers to add CAP_SYS_ADMIN to a binary (like the module does).

But again, just trying to understand if it is a packaging problem, a module problem, or other. The difference between what the module does and what your linked configuration does is running sunshine as root -- the module runs it as a user (either with or without CAP_SYS_ADMIN), which is necessary in many configurations and, in fact, is what upstream does by default (runs as systemd user unit). If you have the same issue just running sunshine & with CAP_SYS_ADMIN as a user without the PR, then it points to a problem not with this PR. Unfortunately not sure how much help I can be debugging this -- hardware encoding works fine on AMD as a user for what it's worth -- happy to help think about it but am not currently able to reproduce on any system I have.

@tymscar
Copy link
Contributor

tymscar commented Mar 31, 2024

Sorry it took so long to reply. I spent some time trying what you suggested.
I didn't know about scurity wrappers before this. After using that on my aformentioned setup, it works great without sudo actually, thank you!

So somehow, somewhere the issue isnt with sunshine or with capsysadmin, it's with this service. Any other ideas of what to try?

@devusb
Copy link
Contributor Author

devusb commented Apr 1, 2024

@tymscar wondering if it's something with startup order -- can you try just restarting the service as
systemctl restart sunshine --user
after you're completely in your desktop after a fresh boot and see if that makes a difference?

also, do you have hardware.nvidia.modesetting enabled? https://search.nixos.org/options?channel=unstable&from=0&size=50&sort=relevance&type=packages&query=nvidia.modesetting might also be worth a shot.

@tymscar
Copy link
Contributor

tymscar commented Apr 2, 2024

@devusb So I think I figured it out. Took me like 15 hours of nonstop trying everything I could to get here, so let me explain.

Out of the last 10 commits to nixpkgs related to sunshine, only 4 of them actually work, and they arent the last 4, or contiguous.

Now those 4 work great when standalone, but when ran as a service, this is what it looks like:
image

Took me ages to figure out, but apparently the offset of the colour space, and the image being cut in half with my right side of the monitor on the left, happens because I dont use my monitor's resolution on my phone when connectiong. If I set it to that, it works great. Im not sure why this only happens when running as a service, but by this point I am just happy I figured it out.

AV1 encoding with nvenc is fine as long as I tell it to run with CUDA.

@Cryolitia
Copy link
Member

Cryolitia commented Apr 9, 2024

I would like to divide it into two module: programs.sunshine and services.sunshine. Some people(like me) may only want to start it manually form terminal unusually, instaed of a systemd service or programs.sunshine and programs.sunshine.autoStart like

autoStart = lib.mkEnableOption (lib.mdDoc "Clash Verge auto launch");

Fair ask -- this is not my use case, unfortunately -- not sure I'll have time to add/test it here. Could certainly be added with a later PR though.

At least please use programs.sunshine , or the later PR should add an alias for ranaming services.sunshine to programs.sunshine

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/3793

@SuperSandro2000 SuperSandro2000 merged commit 5bfe2b6 into NixOS:master Apr 20, 2024
24 checks passed
@devusb devusb deleted the sunshine-module branch April 21, 2024 00:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog 8.has: documentation 8.has: module (update) This PR changes an existing module in `nixos/` 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10
Projects
None yet
Development

Successfully merging this pull request may close these issues.

NixOS module request: sunshine
10 participants