Support single audio mix server mode to support large ensembles and simplify the mixing #599
Comments
|
@storeilly, @kraney, @maallyn I updated the code on the feature_singlemixserver branch with the following functionality:
It would be great to get feedback from you:
|
|
Will each client be able to balance their own input in the mix?, if not I can imagine one of two scenarios. |
|
I really like this feature since it has a huge potential. If really only the "master client" can do the mix I don't see any problem @storeilly? What I see as a problem is that the clients just see "TRYING TO CONNECT". I'd prefer (at least for newer clients) to show "Please set the audio quality/... as follows: [...]" or even a server message which sets them automatically. If you don't do that it would probably rise quite some support questions and problems. |
|
I don't like it. Sorry. Consider human nature and the fact that you are
removing an element of control from them. If they can't adjust their own
perception of their volume in the mix without affecting how others hear it,
they will compensate and use the control that they have which does. They
will sing too loud or too quiet (depending on personality type). Any
attempt by an outside force (the director) to balance the mix will force
them to increase the compensation to detrimental levels.
…On Sat 19 Sep 2020, 13:16 ann0see, ***@***.***> wrote:
I really like this feature. If really only the "master client" can do the
mix I don't see any problem @storeilly <https://github.com/storeilly>?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#599 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABIJSQLRZPBGA6P3VVZFURLSGSOI7ANCNFSM4RS3AX3Q>
.
|
|
Is there any way what's done could be tuned to having each participant (apart from the director) see only their own channel control? This wouldn't save server side processing - you'd still have n mixes to create. But it retains simplicity in the UI whilst preserving control. The levels would be set by the director, apart from a participant's own level, which they could set for themselves. Perhaps it would still need a "balance" so you could raise your own level against the rest of the mix, or in this mode, your own slider becomes "balance" -- at the bottom, you only hear everyone else, at the top, you only hear yourself, in the middle you hear the director's mix. (I'd also think a "list participants" option would be needed. Otherwise I'd be wondering "who's here?" all the time.) Of course, my other question is -- how does this affect the jam recorder? Is the emit AudioFrame still done before the mix? This seems like it would answer one of my questions on the thread for recording a pre-mixed file - the director's mix is what you'd record. (Though I'd still advise a new signal/slot, it needs the filename handled properly and the file not included in the projects.) |
|
To me the scenario that leads to clipping sounds pretty passive-aggressive and like it could best be resolved by the director mentioning out loud that he’s having to dial down (or up) someone’s volume. It doesn’t seem like an indictment of the entire feature. I haven’t had a chance to try it yet for time zone reasons but I should be able to run it during this weekend. I have a little concern with the “first to connect” solution for selecting a director only because, supposing it goes wrong and someone accidentally joins before the director, how can it be resolved? Everyone has to leave and then wait for confirmation by the director via some other channel before rejoining? However I understand the desire to minimize the disruption to the code. One other idea for selection comes to mind - maybe the director is the first one to join who chooses a baton as his instrument? |
That may or may not be. I think this strongly depends on how the singers can adapt to the mix. For some it may be simple to adapt, for some it may not be possible to sing. But this can only be evaluated on a real test with a lot of singers, I guess.
Sure, a lot things could be done. The current code implements the basic functionality. If it turns out that this mode is useless, all additional effort would be wasted. So the first step is to verify that this mode works with a lot of singers (which are all instructed before entering the server what they have to set in the Jamulus client). If that was successful, more time could be spent to improve that server mode.
Yes, exactly. If you want any individual mix, you have to use the normal mode.
The server could send a Chat message when a new client enters the server. But as I wrote above, before any more time is spent, it has to be verified that this new mode works in a real scenario and turns out to be useful.
The jam recorder is not affected. It works the same as in the normal server mode.
No problem. You can simply swap the places at the server. The one who currently has control leaves the server. Then the director leaves the server and immediately re-connects to the server. Now the director has the control. |
|
The swap sounds simple for a small group, but really problematic with a group of 60 people who are all joining a rehearsal that is starting, each according to his own timing. But I do want to say I really appreciate the very fast turnaround on this, and the approach to try with the simplest change first then refine once the principle is demonstrated. |
|
I ran a test using the current master first to serve as a baseline. For my test I connect once using a normal client I can listen to, then add a bunch of headless clients that are just silent and produce no audio out. (Connected to a jack dummy audio.) Because of this I can't be 100% sure that they work equivalent to a normal client. However I did check that if one of those is "master", I can connect successfully with a normal client and get an empty mixer, but hear audio. I'm using 2vCPUs on a cloud server. Intel Skylake architecture I believe. In this test I was able to connect 62 clients, but the last few didn't go smoothly. The user name took a long time to appear after the channel appeared. The sound quality degraded somewhat. CPU usage was around 75-80% out of 100 on a two core system. That is, about 160% usage out of 200 in the linux way of reporting it. I then ran a test using singlemix. With this, I was able to connect 80 clients with no immediate issues. Sound quality was good. CPU usage was around 46% out of 100, or about 92% / 200 in the linux way of reporting it. However, it only stayed stable for about a minute. After time passed, without any new clients, it shifted into a very degraded mode. Sound came in short, regular choppy bursts, about half on and half off, like 120 bpm pulses. CPU usage dropped to about 13%. Stopping the synthetic clients does not immediately resolve it, but it does go back to normal once the server times the clients out of its list. This was in normal quality, 128 byte buffer, Mono. |
|
There's one longer term use case I'd like to support that might suggest some requirements for this mode. I'm looking into the ability to re-transmit the Jamulus audio into a YouTube Live or Facebook Live stream. It seems pretty straightforward to do this using SoundFlower on a mac so you can capture what is coming from Jamulus. In such a situation, the director might likely want to send a click track to the band members to control the tempo and dynamics. So it would be useful to have a second mix not tied to the director one, where that one click track can be muted for the audio that will be retransmitted. |
|
Folks: I just did a compile of the feature-singlemixserver pulled as of about 10 PM Pacific U.S. Time on Monday, Sept 21, 2020. It is on my server newark-music.allyn.com, which is a four CPU dedicated server at Linode in Newark, N.J. I did confirm that only the first connect has the sliders and others dont. However, I did notice the following warnings on the compile output: ===================================================================== usr/include/x86_64-linux-gnu/qt5/QtWidgets -isystem /usr/include/x86_64-linux-gnu/qt5/QtGui -isystem /usr/include/x86_64-linux-gnu/qt5/QtNetwork -isystem /usr/include/x86_64-linux-gnu/qt5/QtXml -isystem /usr/include/x86_64-linux-gnu/qt5/QtConcurrent -isystem /usr/include/x86_64-linux-gnu/qt5/QtCore -I. -I. -I/usr/lib/x86_64-linux-gnu/qt5/mkspecs/linux-g++-64 -o clientdlg.o src/clientdlg.cpp ============================================================== The chat message does show the git log as well as a warning that this server will have faders only on the first connection. This server is on the central server listing for All Genres. Mark |
|
I spent some time poking around with the audio mix code and I've found some changes that improve the maximum scale even without resorting to the singlemix option. Specifically
With tweaks to those items, I have been able to scale to 65 channels on an 8-core system with no notable audio issues; none of the quirkiness I was noticing in prior tests. That's with
Note that I only seem to be able to use about 40% of each CPU core on this system before I exceed the time budget for mixing. It might actually be possible to improve the results by doing the mix for half of the channels at a 0.5ms time offset from the other half, at least for this CPU architecture. That is, don't roll all of the threads into the same FutureSynchronizer. I also tried adding an indexed lookup for channels by hostname in hopes of making PutAudioData more efficient, but I don't believe it made a significant difference. PutAudioData is not the bottleneck. |
|
Better yet, don’t use a FutureSynchronizer at all and have a separate timer loop per channel block. Then |
|
Sounds interesting @kraney. I also consider the OnTimer() triggering is critical to the process (#455 (comment)), as it occurs asynchronous to the audio processing stuff, and I think timer exhaustion (ie decompress+mix+compress loop not finished before next timer triggers) should be logged somehow to get a sense if the system isn't introducing audio artifacts due to overload. Did you implement any probe to detect that case? Anyways, looking at Qt docs (https://doc.qt.io/qtforpython/PySide2/QtCore/QTimer.html#accuracy-and-timer-resolution) I'm unsure how Jamulus reacts if OnTimer retriggers while still active in the processing loop due to overload.
|
That is interesting. With your settings (stereo, 128 samples) we already had a report of supporting 100 clients, see #455 (comment). The question is what was different when brynalf did his test. Certainly, his CPU is faster but 65 vs 100 clients is a big difference.
I would say that this is no practical way. pljones wrote a nice picture about the threading situation: #455 (comment). So we have one part which decodes all OPUS packets and then we have multiple threads which work on that data. If you have different timers, how to deal with the common part at the beginning since each client needs the decoded audio data from all other clients. |
|
Looking at #455, brynalf is on a 32-core CPU which would hide the main things I ran into with a smaller number of cores. idealThreadCount would be 32 already, so it would probably not be a limiting factor. I could run with more cores but it's not really practical for a cloud-hosted instance; it would get rather expensive. And really, the capabilities of the 8-core instance I was using are greatly underutilized. For the common decoded audio data - I would say it seems like it would make sense to preserve the decoded data in a circular buffer. Each independent OnTimer would "top it up" with the latest new data and expire an equal amount of the oldest data, then work on the result. It looks like #455 would be the more appropriate place to continue the discussion - sorry to hijack this ticket. |
|
Or another thought, why preserve the incoming encoded packets at all? Why not decode them on the way in, and store only the decoded data? |
|
It would help if at the very least the next decode task could start before the mix tasks from the prior timer have completed. That would let CPU utilization get closer to 100% per core. |
|
Question on the implementation for this: After the Leader has joined the server, do all other users see no faders, or faders that are inactive, or what? I think they shouldn't see faders that appear to be active, but should still see Profile Names of connected users. |
|
To @storeilly's concerns above, I still think there's huge value for choral groups in this feature. Imagine an in-person rehearsal:
There's no individual mixer controls - everyone hears the group sound "unmixed", then adjusts their individual output to fit. The Jamulus equivalent would be to send unmixed sound to all users (or to ignore their fader settings), then have individuals adjust their gain + singing volume to compensate, which this addresses. That doesn't solve for the user who says "I can't hear myself well enough" or "The group isn't loud enough", but that's not a problem that Jamulus can solve. Jamulus' faders are not a control for System Volume - they're a control for "Jamulus within the bounds of System Volume". So if System Volume = 50%, then setting a Jamulus fader to 50% is equivalent to setting System Volume = 25%, and setting a Jamulus fader to 100% is equivalent to setting System Volume to 50%. If the user isn't getting enough volume from a "flat" mix, they need to increase their System Volume (which could be PC sound card or audio interface output), add an in-line mini amp to headphones, etc. |
Isn't that partially self-contradictory? Each individual has their own mix: it's what reaches their ears, just like in Jamulus. They can control their own input level - no one else has control over that, just like in Jamulus.
That's right. And it's generally how I play. I expect people in the group to use their ears and adjust their input levels so that the group is hearing a good mix. It's strange the difference in etiquette between NINJAM and Jamulus. In NINJAM, it's very much seen as "your problem" if you're too loud or quiet. You're affecting everyone else in the group, so it's down to you to fix it. In Jamulus, there's much more acceptance that everyone has control over what they hear and is expected to handle whatever someone else does -- even though it's affecting everyone in the group. |
|
Thanks you so much for your recommendation!
As this branch is based on 3.5.10, I have to use clients < 3.6.0.
I use the Raspberry JackTripFoundation-Image and they offer a really easy web interface to handle the login. I can register my (private) server and invite all users. Only the mic gain and output level can be adjusted, no much possibility for mistakes... |
|
Hello and thanks for Jamulus in general! I like the "delegation" idea a lot, should that become a separate ticket? In my dreams, this would include downstreaming the slider positions, so I can derive my personal from the conductor's mix. (edited:) I just found that is just another step from there:
All clients including or excluding the serverinstance? |
I have just merge the latest code into the feature_singlemixserver. Can you please test if it now works fine for you? |
|
Wow, that's great. I will test on Sunday. |
|
I've just tested singlemixserver branch after merge with master and it works the following way:
This behavior is independent of the version of the server instance, works with server version 3.5.10 and 3.6.2. Is there a way, to overcome this restriction in the singlemixserver branch, because this does not apply to master branch? |
|
Hi fellow musicians! I created a new rudimentary solution to the
I implemented this for our mixed choir of around 25 people. We start our weekly rehearsals with a soundcheck where previously everybody had to create a good mix (which is hard and time-consuming). Now our conductor can do it for all of us which should improve the sound quality a lot. It's paramount to us that we don't hear Jamulus monitoring. Most of us don't use either no monitoring or local monitoring (from their interface). Would be glad to have some people test this! You don't need custom clients, just a custom server build. Next steps to make this feature-proof:
Happy to hear any feedback! |
|
Great! Thank you very much!
It would be great if this could be controled on the client side. (To be able to follow rule one).
For long term: I think this should not be the case (but for a PoC that's ok): Hide all the level meters/group/mute/pan/... and give a prominent message: "Single mix enabled". |
It would be great if the clients could have disabled controls that show the mix that the master has decided. |
Excellent feature, thx for the implementation. What happens if the conductor (first client) disconnects? (for any reason). |
|
Hi all! I got a little update (now on a different branch on my fork):
This could be done by not disabling one's own mute / fader control?
That's a good question and needs to be tested. In my understanding, the first one to connect to a server get's the channel ID 0 and keeps it for longer than he's connected. After disconnecting and connecting I still got ID 0 and it was not passed on in the meantime. If that's correct, I suggest to keep it that primitive right now, since all other solutions of authorizing/delegating the 'master' role seem much more complex and error-prone to me. Unless somebody has a really good 'simple' idea? :)
Option A: The master sets only his own mix ( Option B: The master sets everybody's mixes on the server (so the server would actually update for ( j = 0; j < iNumClients; j++ )
{
// get a reference to the audio data and gain of the current client
const CVector<int16_t>& vecsData = vecvecsData[j];
{
float fGain = vecvecfGains[iChanCnt][j];
if ( bSingleMixServerMode )
{
// overwrite gain with the gain of the master client (same mix for everybody, except ...)
fGain = vecvecfGains[0][j];
// mute people's own channel (we don't want them to hear themselves in the common mix!)
if ( iChanCnt == j )
{
fGain = 0.0f;
}
}
}Would really appreciate somebody testing the current implementation, at least server-side (so without any visual hints on the client side, but same audio behaviour). Cheers! |
|
Great thing, I'm happy that there are some new ideas and implementations for the chorus branch of Jamulus! I'm using the singlemixserver branch for rehearsals in my chorus since two month - it works great. We are around 40 singers - it works stable and smooth. But in our setup, all participants - except the master - are using a Raspberry PI with a VirtualStudio - image from JackTrip. It has a very basic Web-UI - perfect for singers with basic PC knowledge and for all not to be deviated from the sheet music. |
|
I need your input! I am still not sure what the most "Jamulus way" is to tackle the Jamulus-routed monitoring: Method 1: Always mute people's own channel in the single mixes (this is currently implemented on my fork) Method 2: Let non-masters control at least their own channel in the mix Method 2 gets even more nasty when you think about this: Who controls the pan/gain of the master herself on the common mix? If the master controls it for everybody, she effectively cannot mute herself (which wouldn't work for our choir's conductor). But if the client controls the master's gain/pan, the UI gets even more complicated: For non-masters, all controls will be disabled except for both one's own channel as well as the master's channel. This is pretty complex for a user to understand imho. In terms of KISS I would prefer Method 1. Also, in our choir use case nobody relies on Jamulus-routed monitoring. What do you guys think? |
|
The problem is that the "official" rule is to listen to the signal from the server. What about adding a Parameter to the single mix Parameter which enables/disables the own signal for all singers? |
|
I read in some threads that Jamulus monitoring is a must to be able to guess the delay and to hear my voice with the same delay as all others. Otherwise, I get the impression, that I'm faster than the others if I switch off Jamulus monitoring. It is mentioned as a source of slow down the speed. |
Can you elaborate on that one?
That's a good extension to Method 1 I think. We could have
In my personal (choral singing) experience, this is not an issue at all. Rather, I am barely able to speak a few sentences while hearing myself with the overall delay (of let's say 50ms) without going crazy. :) But use cases might especially differ for non-vocal instruments and so I don't want to just rule our use case for everybody (i.e. never use Jamulus-routed monitoring). So as @ann0see suggested: Have |
|
Better have For the official rule see the getting started page on jamulus.io and Volkers' paper: https://jamulus.io/PerformingBandRehearsalsontheInternetWithJamulus.pdf |
|
I'm done with the GUI changes and created a (to-be-reviewed) PR here: #1381 Suggest to continue the discussion over there :) |
|
Locking this issue now so as to keep discussions in fewer places. See also #1380 |
|
Hi, Sorry for the maintenance noise here. I’m just into triaging issues. Since this issue is locked, and another discussion is linked by gilgongo it doesn't make sense to leave this issue open. Therefore, I'll close it. Please continue any related discussion in the other discussion. Note to the topic: See the feature branch: https://github.com/cwerling/jamulus-mastermix/tree/feature_singlemix_reloaded which any interested person can take up if needed. Please open a new issue if you plan to improve it. |
See here:
A simple approach with only little changes in the Jamulus server code is wanted. Specification:
Adding a new command line argument to the server like --singlemix:
There is a vecvecbyCodedData buffer which is used for both, encoding and decoding. I'll introduce a separate buffer so that I can re-use the output buffer of the first client for all other clients. So instead of calling MixEncodeTransmitData for all the other clients, they simply get vecChannels[iCurChanID].PrepAndSendPacket ( &Socket, vecvecbyCodedData[iDirectorID], iCeltNumCodedBytes );. I just did a quick hack: If I modify CreateChannelList so that no client is added, the audio mixer panel is just empty. This would be the case for the slave clients. But then they do not see how many clients are currently connected which is not a big issue. If "--singlemix" is given, "-F" and "-T" is deactivated and a warning is shown that these cannot be combined. In the OnNetTranspPropsReceived function we can check that the client uses 128 samples, if not, refuse to connect.
There is a new branch were the implementation is done:
https://github.com/corrados/jamulus/tree/feature_singlemixserver
The text was updated successfully, but these errors were encountered: