4008: Rate limited #20
Comments
|
Interesting, there could be a concurrency issue between discordSendPCM and fromMumbleMixer functions. My theory is that there is some thrashing between Discord speaking states (true and false) which could trigger the rate limiter. If you are willing to do some debugging I pushed a separate branch with some debug messages added. If I'm right we would see a high cycle rate in the log. Here is some output from my local dev.
You can see that there are few instances in which the timing is not quite right and the speaking state "flickers". If this doesn't yield any insight on the weekend I can try replicating your env. |
|
That sounds like a good theory. With your debug branch there is a lot of activity when I say, "Can you hear me?" No wonder it sounds choppy. |
add silence to -> discord streams as suggested by disocrd docs closes #11
add silence to -> discord streams as suggested by disocrd docs closes #11
|
@namtsui I have pushed another commit to the issue-20 branch which implements configurable buffer size. While I can't quite reproduce your issue the buffer eliminated any extra Discord speaking state transitions. A 50 ms buffer seemed to be more than enough for my dev setup. I would suggest trying a larger buffer size if you see the speaking state cycling even though mumble transmission is contious. In addition, I added frames of silence on state transitions recommend by the Discord documentation. This should increase the quality of the interpolation if everything is working fine but if the timing issue occurs my best guess is that these frames will stack up in the buffer and reduce the cycling. Give it a try and let me know what you think. If things are not quite right on your end I could try introducing some fake packet timing issue on my dev env to see if I can get something similar to what you saw. A reference for me could you describe the hardware you are running the bridge on? uMurmur? The network connection between them? |
|
-to-discord-buffer 50: flickers frequently it seems like I can't hold down for 10 seconds and speak. In the 250 output, you can see me trying to speak for 10 seconds but it gets broken up. delay 50: delay 100 delay 200 |
|
bridge is running on a vultr.com vps with 512 MB and 1 core. I actually am using murmur instead of umurmur because I don't want to reboot the vps and I couldn't figure out how to relaunch umurmur. I have 70 ms from my mumble client (linux) in california to the bridge/mumble server (openbsd) in florida. One thing I haven't tested is bridge + mumble server on my LAN. Let me try that. If that is still choppy, it seems like it is likely either a firewall issue or an OS issue. OpenBSD does use 100 ticks timing for the kernel so that may affect things. |
|
I initially tested mumble (linux) and bridge/umurmur (openbsd) on my LAN. It experienced the same issue as the vps with choppiness no matter what the buffer is.
Using 100 ticks is the problem. When I compile a 1000 tick kernel for openbsd, it works! It is smooth, and I can hold down the microphone for 10 seconds even with the default buffer of 50ms. Perhaps this can give you ideas about some timing issues in the program? |
|
Awesome glad to see it's working. The audio passed through series of goroutines that handle steps like mixing and encoding. There is variability in the length of encoded data which complicates the matter. I use the least common denominator of 10 ms of audio to pass around internally. When sending to Discord I need to stack up two internal messages into 20ms of audio before encoding and passing it to the discordgo library. The timing issues are twofold. Internally we tick at the audio rates so it's possible to mix and send data at an appropriate rate if packets do not arrive in time it can result in poor audio quality especially when mixing. Buffers tend to help with this. Second, the scheduling and the evaluation of sleeping goroutines play an important role in how latency and jitter of the audio passing through the bridge. At the moment I'm counting on the internal scheduler to be sufficiently fast to ensure each step evaluates at approximately the right time. I put together a little image that shows the flow of audio. I know the bridge is far from perfect. While putting this diagram together I had several ideas on ways to improve the flow. I am not familiar with OpenBSD and the tick rate you mentioned. Any idea on how that kernel setting would affect the networking stack? How long processes can sleep? Go process scheduling goroutines? |
|
here is a good article regarding how to enable 1000 Hz clocks. https://openbsd.amsterdam/clock.html I've come across this problem before in some games where the 100 Hz clock affected the framerate. Since it is 100 ticks / second, that's a resolution of 10 ms. You won't be able to get 1 ms resolution, so that might explain the bad performance. I do not know golang but I looked up maybe you can tell me if my interpretation of your code is correct. mumble-discord-bridge/discord.go Line 72 in a18df28 This line reads from the ticker's channel and blocks. Once the next tick (initialized to 10 ms) is available it dequeues the time from the channel and doesn't make use of it. So the purpose is to kind of slow down the infinite for loop while allowing for channels to communicate (as opposed to sleeping)? So this is a common go idiom? https://golang.org/pkg/time/#NewTicker
There's nothing receiving this right? In any case, normally on linux systems, say, you might be able to get a tick at 13 ms, but on openbsd it would round up to 20 ms, resulting in poor performance. (Linux is tickless though apparently.) |
|
There's no guarantee on the tickers, too, as it seems like it is a minimum. So if you since you set it to 10 ms, but it happens at 13 ms, we get punished because we must wait until the next openbsd tick at 20 ms. |
|
@namtsui I built a ticker that will minimize drift. I noticed an individual that had a long mic on time that was causing the buffer to fill up on my test server. The new ticker is designed not to drift even if the system is busy for several ticks. It's a little less efficient but I did a pprof side-by-side and I'm not concerned with the additional overhead. I'm curious to see what running "go test" produces on your hardware. As a reference, I took a screenshot of the result on my dev machine.
|
|
default (100 tick) openbsd kernel: custom (1000 tick) openbsd kernel: |
|
That first result 😕 Gald to see the new ticker improvement is replicated. I set up OpenBSD in a VM tonight and got a similar result. (I have not tried the BIOS tweak) I ran into issues building in OpenBSD which ultimately resolved by editing line 5 of https://github.com/layeh/gopus/blob/master/opus_nonshared.go to include OpenBSD. How have you been building and running tests? I'm thinking that would trough the ticker out and just put the goroutine to sleep using the math I implemented in the ticker. That would reduce the need to spawn goroutines and block using channels. Will try it tomorrow night. |
|
I ended up just building it as a user ( At first I used a golang port generator for openbsd, which tries to automatically create a golang port. see: https://www.namtsui.com/cgi-bin/cvsweb/ports/audio/mumble-discord-bridge/ However, I have to manually patch Then when I run |
|
Ok, so I tested it by replacing the ticker with a sleep implementation. Here are some new results. Linux OpenBSD in VM Some conclusions:
It might be best at this time to simply suggest that the users tweak their kernel to a faster clock. To close this issue off maybe we update the README for any future OpenBSD adventurers and bring in some of these updates implemented. Any other thoughts? ideas? suggestions? Any code you would like to contribute back? |
|
Something like this could be added: "OpenBSD users should consider compiling a custom kernel to use 1000 ticks for the best possible performance." If I submit it as an OpenBSD port in the future I can also note this in either the package MESSAGE (printed when installing a package) or README. I don't really have any other input. It is nice to hear that setting out to solve one problem uncovered another more serious problem. |
|
I guess I didn't fully test it. mumble --> discord works. However, I never thought to test the converse until now. discord --> mumble throws errors and nothing is audible. Similar to: #4 (comment) |
|
it also eventually segfaults. Here is a backtrace without debugging enabled. Maybe gopus or gumble related. |
If I hold down the push to talk key on mumble for 20 seconds I will get rate limited by discord and the bridge disconnects. I also get rate limited and booted off if I speak in bursts normally. I also noticed that the discord bot blinks repeatedly on discord.
Possibly related:
bwmarrin/discordgo#830
bwmarrin/discordgo#829
umurmur hosting mumble on openbsd
bridge running on openbsd
The text was updated successfully, but these errors were encountered: