Support large ensembles (> 100 connected clients)

[jp8]

I would like to open a discussion about improving the Jamulus user experience for large ensembles. My understanding is that the current Jamulus server will use only a single CPU core, and that it generates a personal mix for each connected client.

One potential solution could be a server mode in which a single mix is generated, then potentially the server would have less work to do, and could therefore handle more connected clients. I image the client who occupies the first space on the server would be in control of the mix for all participants.

A second potential solution would be the ability for a server (with mixer controls on the server UI) to also act as a client to another server. In this case all the violins could join server A, all the cellos could join server B, and servers A and B could join server Z. The conductor would connect his client to server Z and have a mixer control for each section. In this solution, larger ensembles would simply require more servers. Delay would be mitigated by having multiple servers at the same hosting centre, even indeed on the same multi-core VM, so the ping time among all the servers is 0.

A third potential solution would be to have the server use multiple threads to generate mixes in parallel.

I would appreciate hearing what people think of these approaches, and I would like to hear about any other approaches that people can think of.

[corrados]

How many clients do you have to support? Have you tried it out already?

[jp8]

I would like to support 120.

And I am trying to think not just about the server capacity, but the client user interface also.

So far the largest group I've been in is 30. The server had 8 CPUs but my impression is that one of them was at 100% and the others were mostly idle. The user interface would have been improved if it was a grid instead of a single line.

[corrados]

A third potential solution would be to have the server use multiple threads to generate mixes in parallel.

This is the only meaningful solution. The OPUS encoding requires the most processing time. This could easily be calculated on different CPU cores.

I would like to support 120.

The biggest challenge in that case is support all 120 musicians to setup their Jamulus clients correctly. I guess most of them will use ASIO4All together with the laptop built-in sound card. This will give them bad latencies and they will not have much fun playing together.

And I am trying to think not just about the server capacity, but the client user interface also.

Are you sure every of the 120 musicians will take the time to adjust all 120 faders? I don't think so. If you have so many musicians, each of them will have to adjust their input volume so that they all have about equal level. Then there is no need to touch any fader.

[corrados]

I would like to support 120.

If you make it, you should apply for here: https://www.guinnessworldrecords.com/business-marketing-solutions/record-event-formats/online-records ;-)

[jp8]

Are you sure every of the 120 musicians will take the time to adjust all 120 faders? I don't think so. If you have so many musicians, each of them will have to adjust their input volume so that they all have about equal level. Then there is no need to touch any fader.

Indeed one thing i liked about the 'multiple server' approach is that musicians would only see faders for the others in their section, a section leader would control the section's mix that got sent upstream to the conductor's sever, and the conductor would see only one fader per section. Nobody would ever need to deal with 120 faders on a single screen.

[corrados]

multiple server' approach

That is no good solution in my opinion. You'll add additional latency and you will also have problems with the synchronization. You need a single point where all the audio streams are mixed together, otherwise you would have to compensate for the different delays. One single server is the way to go here.

the conductor would see only one fader per section

There exists already a similar feature request: #202

[jp8]

That is no good solution in my opinion. You'll add additional latency and you will also have problems with the synchronization. You need a single point where all the audio streams are mixed together, otherwise you would have to compensate for the different delays. One single server is the way to go here.

What if the up- and down- stream between servers was uncompressed audio and didn't go through Opus? Gigabit ethernet in the hosting centre should be able to handle that, I think.

[corrados]

didn't go through Opus?

I am sure with using the OMP and with 8 CPU cores in your server you will be able to serve 120 clients.

[jp8]

Yes you are right about the multi-processing on the server.
It leaves me wondering if there is a way to eliminate the faders at the client (maybe show just a grid of 120 names) with a way for a single sound engineer with a high-resolution screen to adjust a single mix that is heard by everyone.

[corrados]

adjust a single mix that is heard by everyone.

No, that is not possible. Each client has it's own mix. And I think that is useful since in a real orchestra you hear the instruments which are close to you much louder than the others which are far away from you. In your personal mix you can configure the same.

[sthenos]

Just for your information during the world Jam on Saturday I bought a dedicated Google VM with 8 cores and 16GB RAM plus 200GB SSD. We were able to reach about 35 people on the room all jamming together, server was running at 110% cpu utilisation across all 8 cores, but the server didn't crash as it had done on the 4 core version I was running the previous week at 28 people.

[corrados]

That is very interesting. The main Jamulus processing routine is a single CPU core implementation. There is no multi-threading implemented yet. Now the question is why you are seeing 8 cores busy when the Jamulus server runs. Maybe the Google servers do something smart with the running applications? Maybe they split the work themselfs somehow and distribute it on all available processors. But if this is the case, how do they do it?

Also it could be that simply the CPU monitoring tool shows incorrect data...

Anyway, having 35 clients connected to the Jamulus server is very impressive :-).

[corrados]

Another input: This user reports with 35 connected clients a CPU usage of only 17%: https://sourceforge.net/p/llcon/discussion/musicianslounge/thread/4702d9fae1/#86b5

[corrados]

The user interface would have been improved if it was a grid instead of a single line.

What about a "slim design" of the faders:

[JimMooy]

So here is my screen shot with 35 connected:

The audio sounded good for the local area clients. Only 10 to 15 people were actually making sounds. I am trying to get 50 clients for next week's test of a Linux server on a 20 up 100 down business modem.

PS: You are a fast reader Corrados. :)

[corrados]

I did not know that you have a Github account and read the Issues. Thanks for your screen shot. I have modified it by making all the IP addresses invisible (for privacy reasons).

I am trying to get 50 clients for next week's test of a Linux server on a 20 up 100 down business modem

Please report here when you have done this test.

[JimMooy]

Oh thank you! That was a NEWBIE in action. Much appreciated. I'll let you know how it goes. Looking to get all windows users off of ASIO4all. Nothing but problems with ASIO4all for the novice users. I conduct a 70 piece community college orchestra and a 25 piece big band. We (and every other music educator in the world) are wondering how to rehearse our groups when school begins again in August. Thank you for your hard work.

[jp8]

I suppose this is a 8-core cpu. the 13% usage for Jamulus represents 100% of a single core. On a linux server, under a similar load, 'top' would report 100% usage (out of a total possible 800%).

[jp8]

The user interface would have been improved if it was a grid instead of a single line.

What about a "slim design" of the faders:

It's good but what about this:

• for each instrument icon, a a column (a vertical list) of names, with a single green light (just to the left of the name) if the person is making sound. If there are 9 violin players on the server, then the violin icon has a list of 9 names above it.
• each column could be 'opened' and 'closed' to see the faders inside the group. when 'open', it looks just like the current UI.
• the columns would default to the 'opened' position until there are too many faders for the client's window dimensions.
• sorting by instrument type horizontally, and by musican name vertically

[corrados]

Yes, this would be a possible solution. But I am more a fan of "keep it simple and stupid" and prefer little incremental changes to support a new use case. The "slim fader" would be much easier to implement and be a straight forward change in the Jamulus software.

[corrados]

I suppose this is a 8-core cpu. the 13% usage for Jamulus represents 100% of a single core.

Let's see what JimMooy reports when he has finished his second test. @JimMooy Maybe the next time you should also make a screen shot of the individual CPU cores load like this:

[jp8]

Yes, this would be a possible solution. But I am more a fan of "keep it simple and stupid" and prefer little incremental changes to support a new use case. The "slim fader" would be much easier to implement and be a straight forward change in the Jamulus software.

Yes and for small screens or large groups, it will be a very welcome improvement. Do you think we could have the musican's initial (the first letter of their name) instead of the instrument number?

[corrados]

instead of the instrument number?

Are you referring to my screen shot? The number in my screen shot is actually the name. I just used a number as an example but you could use any number or letter there.

[corrados]

This is what I have so far:




It's not ready yet but looks promising to be included in the Git repo.

[corrados]

I just added the code to the Git master. If you have the possiblity to compile the code and want to test it, you can do it now.
BTW: I called the new skin "Slim Channel". I am no native speaker. Would that name be ok or should we use a different name for it?

Here is a screen shot of the new implementation:

[jp8]

I think you should keep "Slim Channel" as your rap name :)

It looks great on Mac and Linux. Could be slimmed down a bit in the future by not showing the full-length name. And... I see the icons vary quite a bit in width.

[WolfganP]

Good work! I would name it "Compact Channel View" or something similar. Also, for this use case it's probably valuable to force the channel widths to the minimum (ie in this case, all forced to a width like Eli, Vik or V in the screenshot above) and if the name is longer, show the details via hover/tooltip

[WolfganP]

is there any way to run the client in a way that no live input is needed?

There is always a way ;-). But right now this is not implemented/supported in the Jamulus software.

I was thinking something like a pseudo-device File to select in the Settings menu, and open a wav/mp3/whatever is easier to implement from there and stream it in a loop once upon connection.
Or is it too difficult to develop?

[corrados]

I do not think it makes sense to implement this. When I do my multi-threading tests, I am running multiple Jamulus instances under Windows which works just fine for me.

[corrados]

As written in #375, if I use #pragma omp parallel for in the OnTimer() function of the server, the CPU usage jumps to a high value even if the for-loop does nothing time consuming. Is there anybody who can help out here?

[jp8]

I just checked in a new version in Git. The high CPU usage seems to be still present, but the garbled audio should be (hopefully) solved now: db7a759

Confirmed, the audio is good now, for me.

[WolfganP]

I do not think it makes sense to implement this. When I do my multi-threading tests, I am running multiple Jamulus instances under Windows which works just fine for me.

OK, sounds fair. Do you mind to detail how you test with multiple clients so we can help testing and reproduce? Do you use the same audio source for all clients? Each client a different profile or just a diff --clientname?

[corrados]

Yes, it is the same audio source. That was ok for my tests because I was only interested in the CPU usage when I was working on the OMP implementation.

Do you mind to detail how you test with multiple clients so we can help testing and reproduce?

Well, basically I just started the Jamulus client multiple times. That works with my ASIO driver, fortunately.

[jp8]

Some observations with the multithreaded server, on two different laptops.

With just one client connected:

With 4 cores, there are 5 processes
3 are in status 'Running', with CPU around 65-70% each
2 are in status 'Sleeping'

With 8 cores there are 9 processes
7 are in status 'Running', with CPU around 65-70% each
2 are in status 'Sleeping'

On the server with 4 cores, I had 10 people on the server. CPU was around 295. I suspect that was 3x95% but I wasn't watching closely.

[corrados]

Thanks for the info. Next step is to find out how to reduce the OMP overhead to get the CPU load much lower. Let's see if that is possible...

[storeilly]

Hi Volker, I have a chain of private servers for choral use on AWS and GCP and am wondering...
Would it be possible in the interim to link 2 instances on the same physical machine in code and hope the latency between the instances is low enough? In the hope that the instances use different cores. I have a vested interest in this issue as one of my choirs is 75 members due to start back in September.

[WolfganP]

Thanks for the info. Next step is to find out how to reduce the OMP overhead to get the CPU load much lower. Let's see if that is possible...

Maybe the solution is to identify a higher loop/fork point earlier in the process, and let every thread process it's own timer (or maybe use the OpenMP Task directive somehow?). That way there's the fork/join thread creations/destructions are just processed once per server session.

[corrados]

Would it be possible in the interim to link 2 instances

See my above comment: #339 (comment)

[corrados]

or maybe use the OpenMP Task directive somehow?

I am looking for a simple solution. Are you experienced with OpenMP Task directives?

[WolfganP]

or maybe use the OpenMP Task directive somehow?

I am looking for a simple solution. Are you experienced with OpenMP Task directives?

Not really, but I'll keep looking for a workable solution on threads reusability and run some tests.

[WolfganP]

@corrados I was looking to make some test cases for multithreads. Could you please create a multithread branch with your changes at db7a759 so all changes/experiments are properly contained if multiple collaborators include code or build the branch for testing?

[corrados]

Sure. Here it is: https://github.com/corrados/jamulus/tree/feature_multithread

[corrados]

Please note: Please try to stick to what I said above: "I am looking for a simple solution.".

Have you tried out the current OMP implementation with multiple CPU cores and a lot of connected clients? I know that the OMP overhead is significant but I would also like to know how good the spreading of CPU tasks is now spread over multiple cores.

[corrados]

I just implemented a scaling of the instrument picture in case Compact skin is chosen to avoid that the instrument picture makes the channel wider:

[WolfganP]

Sure. Here it is: https://github.com/corrados/jamulus/tree/feature_multithread

Thanks @corrados I was looking at the code these past days and thinking about the cpu-i/o load increase when a high number of clients is connected, and wonder if anyone run a profiling of the app on that test case to verify where the critical points are?

BTW, should we move to an specific thread on server performance to discuss specifically everyone findings?

[corrados]

BTW, should we move to an specific thread on server performance to discuss specifically everyone findings?

Just create a new one if you like.

[corrados]

I plan to do further modifications to the Compact view. If the text is too long, I use a smaller font size. I know that this is hard to read but it leads to a very slim channel:

What do you think about this implementation?

[WolfganP]

I like it as extremely compressed UI for specific use cases (like large ensembles), but as the controls' label aren't explicit on the actions anymore, it will be good if you can add hover tooltips.
Would it make sense to leave the Compact view as it is today for (not so) large groups, and name this as Extremely Compact?

[corrados]

will be good if you can add hover tooltips

These hover tooltips are already implemented.

Would it make sense to leave the Compact view as it is today for (not so) large groups, and name this as Extremely Compact?

I would want to avoid adding a new skin for that. If you only have a few musicians connected than you can use the Normal skin.

[corrados]

To make a cross reference: brynalf successfully served 100 clients in his local area network with his 32 logical processor PC using the latest Git master code: #455 (comment)

[corrados]

There is a new experimental server mode in developement to support large ensembles, see: #599.

[corrados]

Support large ensembles (> 100 connected clients) [...] I would like to open a discussion about improving the Jamulus user experience for large ensembles. [...]
A third potential solution would be to have the server use multiple threads to generate mixes in parallel.

With the latest changes to the multithreading code it is now possible to support >100 clients. So the initial request of this issue is solved.

One potential solution could be a server mode in which a single mix is generated, then potentially the server would have less work to do, and could therefore handle more connected clients.

This has been worked on here: https://github.com/corrados/jamulus/tree/feature_singlemixserver.

Of course we still have outstanding issues in that area but these should be discussed in this Issue: #455.

So I'll close this issue now. Please continue the discussion about this topic in the Issue #455.

[memark]

For anyone trying to start Jamulus Server on MacOS with more than 10 participants, this is the command you need to run in your terminal

/Applications/JamulusServer.app/Contents/MacOS/JamulusServer --numchannels 30