Support huge ensembles (> 250 connected clients)

[DominikSchaller]

I would like to discuss ideas on how to support mass singing events with more than 250 clients.

It seems to be very easy to push the limit from 150 to 256, by changing this line in global.h:
// maximum number of connected clients at the server (must not be larger than 256)
#define MAX_NUM_CHANNELS 150 // max number channels for server

The question is how to support even more. The two limiting factors would be CPU and bandwidth.

• CPU could be limited by using the single mix feature.
• Bandwidth would support a lot of clients on a big AWS EC2 machine with a 10 Gigabit network (or even more).

So I guess my question is: Do you foresee any problems with changing that limit? Is there some stuff that needs to be done on a single core and would max out with let's say 1000 clients?

PS: I'm aware that this would be a nightmare to mix manually, so I am looking forward to the auto-mix feature and the single-mix option. There might also be a way to do an auto-pan.

An alternative idea I had involved chaining servers together (see picture below). The servers and the streaming clients would run on the same cloud netwerk so ping is negligible, but this architecture would come with a latency penalty.

[gilgongo]

Do you foresee any problems with changing that limit?

Since bandwidth use is I assume just additive, would the unknown be how CPU and memory usage scales? Interesting to observe how things behave on a server if you set up a large number of machines each running a large number of clients. Say, 10 Windows VMs each running 100 clients sending a signal to the server :-) I think people use AWS for stress testing like this sometimes if they have the money.

[vdellamea]

since there are many Jamulus servers on the cloud, with some effort a coordinated test could be organised. Who already has an account on the cloud could quickly setup a machine for few cents.

[DominikSchaller]

I assume some calculations are done in matrices which would grow by n^2, but I have no experience with the Jamulus code base yet. Hence my question regarding the single-mix mode which might circumvent the need for it.

I can spin up some machines in AWS, and start a 100 clients each, no problem. I might need some help with sending random sound through Jack though.

But my first question would be how much code needs to be changed for this to happen. I'm sure Volker didn't put the 256 limit there without a good reason ;-)

[ann0see]

I might need some help with sending random sound through Jack though.

If you use Linux, I think you can use qjackctl and connect one jamulus client to another one. Not sure how this works with CLI only, but it should also be easily possible (I'd say Linux is easier than Windows). Maybeyou can compile jamulus with JACK support on Windows?

[DominikSchaller]

I am using Linux. I remember that somebody posted here about doing automated tests with many clients a couple months ago. I will search for the comments and try it out.

[vdellamea]

There might also be a way to do an auto-pan.

In my recording remote I also implemented an automix feature with a rough autopan (uniform distribution from left to right, with extremes half of the distance between clients). Regarding the automix, it's done at the end of the session so it is not immediately applicable for realtime.
Autopan with some structured client name might allow placing different choir sections in different zones (though contiguous). I have to revise what I did because at present distribution is alternated, and I do not remember why I did it in this overcomplicated way that does not help in controlling the positions :)

[hoffie]

I think @softins recently investigated some clever trick regarding signal handling which depended on the number of support channels. Maybe he knows more details.

[dcorson-ticino-com]

If I understood correctly the channel definition is recursive. So going to 300 channels exceeds the compilers possibilities.
That sounds like going further will involve deep diving.

[DominikSchaller]

Yes. It's interesting, 250 is ok but 255 is already exceeding it ... to be honest, this is not in my wheelhouse at all, but maybe some other developer likes a challenge?

[JoeSouthern]

I thought I'd take a look just for fun - didn't make much progress but this info might be helpful. You may have already figured some of these things out.

I tried to find where the - don't exceed 256 in the comments originated from, but couldn't find anything in the docs or posts.

If you change - MAX_NUM_CHANNELS - you probably need to change MAX_NUM_STORED_FADER_SETTINGS in the same file. I think I read that it will override the MAX_NUM_CHANNELS settings.

I found that the compile limit was actually 252 - something like that.

There may be two things.

One is that there is a compiler limitation on template recursions (obviously from the message too). I tried to see how to override that with compiler options, but haven't figured that out yet. I found some things in the MS docs about this for - #pragma inline_recursion( [ { on | off } ] ). A couple of related things from that. I couldn't figure out how to implement it though. I'm struggling with the QT VS crossover.

I think this is one problem and perhaps the issue of going over 252 or so.

I'm sure the template could be rewritten to fix this, but I'm not familiar enough. This is the heart of it I think -

template<>
inline void CServer::connectChannelSignalsToServerSlots<0>() { };

I tired to change this line to feed it say 100 at a time, but it didn't appear to work

connectChannelSignalsToServerSlots<MAX_NUM_CHANNELS>();

The second issue I think could be the actual over 256 issue. There may be a couple of things for it.

The slot_Id - it appeared to be a byte (256) at some points in the code - I could be wrong, but the way it was called it may be taking it as a byte vs integer.

Next, in opus_private.h you have a typedef -

typedef struct ChannelLayout {
int nb_channels;
int nb_streams;
int nb_coupled_streams;
unsigned char mapping[256];
} ChannelLayout;

So, not sure if that 256 works into it and I won't pretend to understand the opus code. But I think this has to also change from 256 to 500.

Finally, there are multiple instances of 256 sample rates for buffers etc so they may work into it too.

Just some thoughts that may help.

[JoeSouthern]

Also - to clarify -

I realize that #pragma's are for inside the code - I did try that - the point is that there are some compiler directives to pass to the compiler about this matter. I'm not sure how to implement that as I explained. There are about 3 directives needed if my reading was right.

You can watch this line -

connectChannelSignalsToServerSlots<MAX_NUM_CHANNELS>();

get down to

connectChannelSignalsToServerSlots<2>();

if you change the overall max settings. It's clearly counting down until the compiler gives up.

I think though as I said - this issue will only get you so far. There are some other 256 issues perhaps, but the compiler is the first hurdle.

[ghost]

I think huge ensembles is possible. Here are some observations:

• Jamulus client sound mixing is O(n^2) at the server, and grouping servers will mitigate this problem.
• Jamulus already has some mixer-grouping. Maybe this can be expoited to reduce O(n^2) within a single server.