Screenshare has poor performance due to no hardware acceleration

[MiMillieuh]

Hello,

That's a native issue too but I may have a suggestion to make it work.

When you're streaming on discord (at least on wayland via xwayland or pipewire for now) the framerate of the app or game are extremely slow. But when discord is the focused window it's smooth again.

My suggestion would be to have a toggle (via a plugin or directly integrated) to keep the screenshare preview without having the window focused. (Disabled by default because I don't know it this issue affect windows and mac users. but it's not the case on the native client for windows at least.)

But that's only a suggestion and maybe that's not even what causes the issue... but it's weird that when the window is focused, this issue isn't happening.

[Covkie]

I've noticed only vesktop struggles with terrible screenshare performance (minutes per frame slow). Native discord only experiences random frame drops.

[MiMillieuh]

I've noticed only vesktop struggles with terrible screenshare performance (minutes per frame slow). Native discord only experiences random frame drops.

Well honestly I have the exact same issue on native client...

Using OBS camera works fine tho

[maotovisk]

I'm also not able to stream at a decent frame rate, 60fps can't be reached and it does not even seem to be remotely close to 30 tbh.

[MattyPixels]

Same here, I can't even stream in 30, it's like a slideshow. The issue is not present on the native client.

[Covkie]

The option to enable hardware acceleration is not in the settings, perhaps that has something to do with this issue?

[Vendicated]

no. its on by default

[MiMillieuh]

I've tried to mess with hardware acceleration but it's still a slideshow...

[seyfu-t]

I think the native client for linux sucks because of electron not being able to utilize hardware acceleration? I don't know Electron all too well but maybe, since this project uses Gluon, there might be other ways of solving this issue?

[Vendicated]

we don't use gluon.

[ant0n-0x0000]

it can use hardware acceleration also

[seyfu-t]

we don't use gluon.

I dont know how I possibly miss-read the readme this bad lol

[MiMillieuh]

Honestly I would like to try to fake having the discord window always on focus.
I've noticed that when I click on my discord window and that the stream shows on the client (not showing the black thing to "use less ressources") it's way more smooth.

But is it possible to prevent that screen and function to come ?

[vickyorlo]

Are you sure that's really what fixes it? I've tried streaming things that don't require me to have the game focused (Beat Saber in particular) and the verdict was pretty clear - the framerate started dropping when there was a lot of movement on screen, but it would go back to relatively normal when there was not much happening. Tabbing back to Discord means you're not doing anything on the screen, which would improve bitrates. It seems to me like the upload is somehow getting throttled.

Additionally, the local stream preview window is in no way reflective of the framerates other people watching the stream are getting - the preview can look absolutely fine while everybody else is getting maybe a frame every two seconds.

[MiMillieuh]

Are you sure that's really what fixes it? I've tried streaming things that don't require me to have the game focused (Beat Saber in particular) and the verdict was pretty clear - the framerate started dropping when there was a lot of movement on screen, but it would go back to relatively normal when there was not much happening. Tabbing back to Discord means you're not doing anything on the screen, which would improve bitrates. It seems to me like the upload is somehow getting throttled.

Additionally, the local stream preview window is in no way reflective of the framerates other people watching the stream are getting - the preview can look absolutely fine while everybody else is getting maybe a frame every two seconds.

Well the only reason I think it's related is that when I'm on screenshare and I click on the discord window, everyone tells me that it's smooth now and right as I click away it's not anymore...

So maybe it has something to to with that screen :

[RinLovesYou]

Might be a long-standing chromium bug that we could work around

[RinLovesYou]

It seems this was not, in fact, the issue.

[MiMillieuh]

Ok so I've tested today and it seems to be what's on screen, for some reasons Discord struggle to send it correctly as when there aren't that much movement it's smooth but when it's moving a lot it's a literal slide show (see video)

I also tried my theory about the focused window with this little code in the console :

document.hasFocus = function() {
    return true;
};

But it seems that the preview is smooth even thought the other client receive a slide show... So it has nothing to do with it.

On the video most noticable things are when I start to move the OBS window fast it begins to be a slide show, In TM, when I stay still the ads and flags are smooth but once I move it's a slideshow, also the bush at the end, when I stop it's getting smooth.

On the left that's the original screen (Via OBS) and on the right it's the other discord client (Via Discord, what a friend would see.)

2023-11-19.00-31-41.1.mp4

EDIT : I have a gigabyte internet and I'm in Ethernet so there are no excuses for slow connexions...

[vickyorlo]

That is what I said... but thanks for testing and confirming, nonetheless.

[simvux]

I've been having the problem shown in the video above, basically regardless of which client I use for discord.

The theory I have is that the video isn't being properly compressed. When someone screenshares from Windows and there's a lot of movement then the video quickly tends to become heavily compressed. Looking like 144p or even worse in darker footage. Meanwhile; even though the framerate sucks, the frames themselves seem to be of much higher quality regardless of movement on Linux. So; it's probably two different ways of which whichever layer is responsible compensates for tight bitrate constraints.

[MiMillieuh]

I've been having the problem shown in the video above, basically regardless of which client I use for discord.

The theory I have is that the video isn't being properly compressed. When someone screenshares from Windows and there's a lot of movement then the video quickly tends to become heavily compressed. Looking like 144p or even worse in darker footage. Meanwhile; even though the framerate sucks, the frames themselves seem to be of much higher quality regardless of movement on Linux. So; it's probably two different ways of which whichever layer is responsible compensates for tight bitrate constraints.

I've thought about it but also on windows the quality doesn't degrade that much depending on the connection. Also I've tested it and the streamer (on Linux Vestkop) sends about 0.5MB/S when there is a lot of movement, and about 0.1 - 0.2 MB/S when still.

Honestily I don't think 0.5MB/S is big but maybe we can't send more and we're trying to sending more... IDK.

[oatmealine]

I've been having the problem shown in the video above, basically regardless of which client I use for discord.

The theory I have is that the video isn't being properly compressed. When someone screenshares from Windows and there's a lot of movement then the video quickly tends to become heavily compressed. Looking like 144p or even worse in darker footage. Meanwhile; even though the framerate sucks, the frames themselves seem to be of much higher quality regardless of movement on Linux. So; it's probably two different ways of which whichever layer is responsible compensates for tight bitrate constraints.

I've constantly had the regular Discord client downgrade my stream when it understands my internet can't handle it - this is okay, as I don't get very high speeds at all - but Vesktop never does this, instead opting to support its quality at all times.
Additionally, I don't actually know if the quality settings work? At least, I've never seen them reflected in the stream preview, when regular Discord always shows what's being sent

[MiMillieuh]

I've been having the problem shown in the video above, basically regardless of which client I use for discord.

The theory I have is that the video isn't being properly compressed. When someone screenshares from Windows and there's a lot of movement then the video quickly tends to become heavily compressed. Looking like 144p or even worse in darker footage. Meanwhile; even though the framerate sucks, the frames themselves seem to be of much higher quality regardless of movement on Linux. So; it's probably two different ways of which whichever layer is responsible compensates for tight bitrate constraints.

I've constantly had the regular Discord client downgrade my stream when it understands my internet can't handle it - this is okay, as I don't get very high speeds at all - but Vesktop never does this, instead opting to support its quality at all times.
Additionally, I don't actually know if the quality settings work? At least, I've never seen them reflected in the stream preview, when regular Discord always shows what's being sent

Previews are in both clients smooth, but I can definitely tell that the same low framerate issues happen to me on both clients when looking at what someone watching the stream would see... As for the quality, I can't really tell... That might even be another issue then.

[heliguy4599]

I wanted to join the discussion to say that I also indeed have this problem. My friends say that my streams are very noticeably bad, nearly unwatchable. I'm using an eGPU with my laptop and ran Vesktop upon the eGPU, still no dice. It seems the problem is not nearly as bad when running on my desktop. Oh and to clarify, I'm using GNOME Wayland on both.

[cyber-sushi]

I'll add my experience as well.
I've tried streaming using Vesktop on both my desktop PC (Nvidia GTX 1060) and my laptop (AMD Vega 8), and they run equally slow at "about 5 fps" according to my friends.

However, I've tried other solutions before, like Discord-Screenaudio, and they seem to suffer from the same issue, as opposed to the regular Discord client.
As others have mentioned, even though the framerate is low, the picture quality remains perfect, it never shows compression or artifacts which usually appear in fast scenes on the regular Discord client or on Windows.

I'm using Hyprland on a relatively weak internet connection (~3 Mbps upload).

[queenbiscuit311]

I've tried discord-screenaudio, vesktop, and pipewire-screenaudio all with the same results. Depending on the situation some clients can be better than others, but there's some apps that are so bad that in order to stream them I had to connect to my computer through moonlight and stream that on a windows or mac system. It seems to be the more multi core cpu usage the worse the stream, and i also cant seem to find any solid indication that discord streaming is actually using my video encoder or my gpu in any meaningful capacity

[alexankitty]

I don't really know the code base well enough to definitively say what my findings mean, but looking through the console while having a stream going shows a couple of interesting things.

[UnifiedConnection(default)] updateVideoQuality: {
    "remoteSinkWantsMaxFramerate": 30
}

[UnifiedConnection(default)] BaseWebRTCConnection.updateVideoQuality: resolutionScaleFactor: undefined, encodingVideoWidth: undefined, remoteSinkWantsMaxFramerate: 30, encodingVideoMaxBitRate: undefined, localWant: 100

[UnifiedConnection(default)] UnifiedConnection.setTransceiverEncodingParameters: maxFramerate: from '20' to '30'

My best guess is it seems like the discord end is saying to change the framerate a few times... and I usually see it change between 20 to 30, but then the other issue I'm seeing is that the encodingVideoMaxBitRate becomes undefined, which I think might possibly be resulting in dropped frames?
Looking at btop on my end, while there's a lot on the screen I don't really see the bitrate exceed 3.5mbps, while the maxbitrate is set to 8mbps and the target being 6mbps.
Perhaps there is something else discord needs to be told when building the stream?

There is also another oddity I'm seeing where the video quality suggest that the values provided by the application and user don't seem to be respected. For reference I'm streaming a 2560x1440 screen at 1080p60, but it appears the stream parameters get dropped to 720p30 at 2.5mbps max bitrate.

[UnifiedConnection(default)] updateVideoQuality: {
    "remoteSinkWantsPixelCount": 921600,
    "remoteSinkWantsMaxFramerate": 30,
    "encodingVideoMinBitRate": 150000,
    "encodingVideoMaxBitRate": 2500000,
    "encodingVideoBitRate": 2500000,
    "streamParameters": [
        {
            "type": "video",
            "active": false,
            "rid": "100",
            "ssrc": 101591,
            "rtxSsrc": 101592,
            "quality": 100,
            "maxBitrate": 2500000,
            "maxFrameRate": 30,
            "maxResolution": {
                "type": "fixed",
                "width": 1280,
                "height": 720
            },
            "maxPixelCount": 921600
        }
    ]

Edit: Web discord is also limited to 720p, a good portion of this might be a limitation of Vesktop technically being the browser version of discord.

[Plarpoon]

That's a good find @alexankitty, I actually was told the same thing by people who watch my streams. This issue definitely needs investigating!

Hopefully you nailed the root issue already, gonna look up if I can find something myself as-well!

[moritzuehling]

I'm new to the project, and would love to try to set the contentHint to "motion" for the mediaStreamTrack in addition to the stream settings, but I haven't found a function in vesktop / vencord. This would favour available bitrate to be used for new frames instead of quality on existing ones.

Is there a way to set those?

[catgirlcataclysm]

I'm new to the project, and would love to try to set the contentHint to "motion" for the mediaStreamTrack in addition to the stream settings, but I haven't found a function in vesktop / vencord. This would favour available bitrate to be used for new frames instead of quality on existing ones.

Is there a way to set those?

I've been experimenting with setting not just contenthint, but fixing the entire API to properly raise the bitrate from 2.5kbps to something normal (80) which should in theory fix most of the screensharing issues. On that note I've run out of ideas of how to implement that, so any help would be appreciated lol. contentHint could possibly also be set if the API rewrite was successful, but I can make no promises.

[moritzuehling]

In theory, it should be possible to write a demo page to try out options quickly, using the plain chromium screenshare interface. Just something that encodes a media-track, and immediately decodes it.

Maybe I can try this out, to see how to make not drop frames.

Might help in understanding the right options to set for streams, and see what's possible inside chromium

[catgirlcataclysm]

That may be a good idea, but I do not believe its the encoding/decoding causing the problem. It may be a good idea to create a small WebRTC streaming application to try and test.
It seems to be the bitrate that actually causes the issue.

[catgirlcataclysm]

I'm new to the project, and would love to try to set the contentHint to "motion" for the mediaStreamTrack in addition to the stream settings, but I haven't found a function in vesktop / vencord. This would favour available bitrate to be used for new frames instead of quality on existing ones.

Is there a way to set those?

Oh and I didn't fully answer this, I understand what you mean, but I believe the issue is that the available bitrate is so incredibly low, that not much improvement can even be made with how low it is. Without fixing that, I doubt we're getting anywhere. Do feel free to try though, you may find something we did not!

[SpookySkeletons]

Where in the chromium codebase is the integer to adjust?
I can hard patch it to check

[Vendicated]

i recommend moving this discussion to the vesktop development channel on our discord server

[EmptyVisual]

i recommend moving this discussion to the vesktop development channel on our discord server

How does one actually get access to talk in the dev channels? I only see core and plugin dev and both say I dont have permission to send messages in channel. There does not appear to be a role that's self assignable for developers/contributors and if we have to jump through several hoops just to discuss on discord it might just be easier to discuss our findings on this issue?

edit: unfortunately the current onboarding process for new users in the Discord server feels confusing. New members can assign themselves roles, but these don't provide the necessary permissions to view or participate in essential development channels (had to manually assign/follow the vesktop dev channel because no self assignable role gives that channel) and still dont have any permission to talk in any of them. This design might discourage potential contributors, especially those less familiar with Discord.

[catgirlcataclysm]

to all of you who have been annoyed about this problem, here.
It's solved in this PR.
#489

[Curve]

Closed by #489.

[sim590]

Is it really fixed? My friends say it's always a slideshow when I stream. I select the default settings:

Here's the version of the package I installed on Archlinux:

Name            : vesktop-bin
Version         : 1.5.2-2
Description     : A cross platform electron-based desktop app aiming to give you a snappier Discord experience with Vencord pre-installed
Architecture    : x86_64
URL             : https://github.com/Vencord/Vesktop
Licenses        : GPL-3.0-only
Groups          : None
Provides        : vesktop=1.5.2  vencord-desktop=1.5.2
Depends On      : electron29
Optional Deps   : None
Required By     : None
Optional For    : None
Conflicts With  : vesktop  vencord-desktop
Replaces        : None
Installed Size  : 16.13 MiB
Packager        : Unknown Packager
Build Date      : Mon 20 May 2024 07:51:09 AM
Install Date    : Mon 20 May 2024 07:51:23 AM
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : None

[polluxau]

Is it really fixed? My friends say it's always a slideshow when I stream. I select the default settings:

Here's the version of the package I installed on Archlinux:

Name            : vesktop-bin
Version         : 1.5.2-2
Description     : A cross platform electron-based desktop app aiming to give you a snappier Discord experience with Vencord pre-installed
Architecture    : x86_64
URL             : https://github.com/Vencord/Vesktop
Licenses        : GPL-3.0-only
Groups          : None
Provides        : vesktop=1.5.2  vencord-desktop=1.5.2
Depends On      : electron29
Optional Deps   : None
Required By     : None
Optional For    : None
Conflicts With  : vesktop  vencord-desktop
Replaces        : None
Installed Size  : 16.13 MiB
Packager        : Unknown Packager
Build Date      : Mon 20 May 2024 07:51:09 AM
Install Date    : Mon 20 May 2024 07:51:23 AM
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : None

Are you using nvidia?

[shenk-i]

Same thing happens to me on AMD, my friends say its a slideshow.

GPU: AMD RX 7800 XT
CachyOS

[sim590]

Sorry, I meant to respond before, but I forgot. No, I am not on Nvidia. I run an AMD rx 5700XT. The version of my MESA driver is 1:24.0.7-3 (archlinux).

[NoelleTGS]

Been having the same issue for a couple months, every time I try and screenshare I need to stop it and start it with every different combination of settings until it randomly decides to work. Bitrate (Target) will frequently be stuck at double digits.

[catgirlcataclysm]

Sorry, I meant to respond before, but I forgot. No, I am not on Nvidia. I run an AMD rx 5700XT. The version of my MESA driver is 1:24.0.7-3 (archlinux).

considering you're on arch, do you have the libva-mesa-driver installed?
it's required for VA-API hardware accelerated streaming :)

[catgirlcataclysm]

Been having the same issue for a couple months, every time I try and screenshare I need to stop it and start it with every different combination of settings until it randomly decides to work. Bitrate (Target) will frequently be stuck at double digits.

interesting, do you mind sharing your system specs? (and if on AMD refer to my latest comment to see what may be a possible culprit)

[NoelleTGS]

Been having the same issue for a couple months, every time I try and screenshare I need to stop it and start it with every different combination of settings until it randomly decides to work. Bitrate (Target) will frequently be stuck at double digits.

interesting, do you mind sharing your system specs? (and if on AMD refer to my latest comment to see what may be a possible culprit)

Not on AMD unfortunately.

CPU: Ryzen 7 5700X
GPU: RTX 3060 Ti
Running EndeavourOS with Nvidia proprietary drivers, and running Vesktop through the Flatpak. Anything else in particular that'd be helpful?

[catgirlcataclysm]

Been having the same issue for a couple months, every time I try and screenshare I need to stop it and start it with every different combination of settings until it randomly decides to work. Bitrate (Target) will frequently be stuck at double digits.

interesting, do you mind sharing your system specs? (and if on AMD refer to my latest comment to see what may be a possible culprit)

Not on AMD unfortunately.

CPU: Ryzen 7 5700X
GPU: RTX 3060 Ti
Running EndeavourOS with Nvidia proprietary drivers, and running Vesktop through the Flatpak. Anything else in particular that'd be helpful?

considering you're on Nvidia, the best thing we can do is hope they one day support VA-API, but aside from that, you're at the limits of your software encoder
try streaming a lower res + fps and see if the bitrate target is able to increase at all(it may not though, as a lower res and framerate wouldn't need as much bitrate anyways)

[NoelleTGS]

Gotcha. It's just confusing because like I said, I have to try over and over and EVENTUALLY it stabilizes, seemingly independent of the settings I pick.

[catgirlcataclysm]

Gotcha. It's just confusing because like I said, I have to try over and over and EVENTUALLY it stabilizes, seemingly independent of the settings I pick.

that is a bit odd, but I may have an idea
try running the latest commit, as a few commits back, background throttling was disabled, and it helps with bitrate throttling in the background
it may be that yours is dropping to the background a lot
I highly doubt it, but it's worth a try, as that is a bit odd.

[shenk-i]

I do have to ask, are you guys using wayland or x11? because i'm a bit confused now

[polluxau]

I do have to ask, are you guys using wayland or x11? because i'm a bit confused now

We should be using wayland but some desktops are still on x11 or have experimental wayland support, shouldnt rlly matter tho

[sim590]

Sorry, I meant to respond before, but I forgot. No, I am not on Nvidia. I run an AMD rx 5700XT. The version of my MESA driver is 1:24.0.7-3 (archlinux).

considering you're on arch, do you have the libva-mesa-driver installed? it's required for VA-API hardware accelerated streaming :)

@kaitlynkittyy Yes, I have the package installed with version 1:24.0.7-3.

[regalialong]

FYI, it would be nice to keep the chatter to a minimum because this has a lot of participants that get CC'd for debugging which should be done in IM / on another issue.

For what it's worth, maybe the user should be warned in the streaming dialog that performance may be worse if Electron is going to use a software encoder (either because of VA-API encoding being unavailable or just using Nvidia)?

[Vendicated]

locking since the original issue has been resolved. for further discussion, either open a new issue or better yet join the discord community and talk about it in a thread in the vesktop channel