Central servers: issues and possible approaches

[gustebeast]

The server documentation says that "Default" has been deprecated in favor of "All Genres". Can we merge these two lists in the client?

[pljones]

The reason they were split still exists.

At the moment, the list can get fragmented if too large and, where UDP fragmentation causes routing issues, this can cause the list to become only partially visible.

I'd also note that All Genres is pretty close to full, so something is going to need to be done one way or another.

As a further point, having the client poll a very large list of servers, most of which they won't be using is a waste of resources both on that client and on every server in the list.

[gustebeast]

It feels strange that we would use genre as a means of solving this scalability issue. Normally I would rely on pagination for that. The client only gets 50 servers or so from the server, then if they want they can request the next 50 and so forth.

The problem is that we can't send them the 50 servers with the lowest ping since the client needs to compute the ping for all the servers before it can sort them. Would it be possible to sort by distance instead of ping? The user experience should be comparable even if the ordering is slightly different. We can get/cache the locations using the client and server IPs. This would also get rid of the ordering issues when pings fluctuate, and the time required for the servers to sort when switching genres.

I'm also curious how much of this scalability issue is client vs server (jamulus.fischvolk.de).

[pljones]

Would it be possible to sort by distance instead of ping?

Distance from the server list? That wouldn't help much. To sort per client, it would have to have some way to compute -- on every client request, every two seconds -- the distance between the client and every server. That sounds like a lot of work. And physical distance isn't much use when it's the ping time that's important. I get half the ping to Frankfurt than Cardiff from London. Even Zurich is better than some places in England.

[gustebeast]

Sort per client yes. We wouldn't necessarily need it to update every two seconds though. The initial request (to central server) would just run once (with a refresh button to re-query). It would require looping through every server, but if you can cache the IP -> location mapping, then you'll just have to do a bit of basic math per server per request: https://stackoverflow.com/a/27943. You may even be able to avoid checking every server if they are stored in a data structure that is organized by location.

Once the client has the 50 servers, it can poll them every 2 seconds for the ping/musicians.

For the Frankfurt vs Cardiff example, my hope is that all those servers would appear near the top of the list, so while the lowest ping may not appear first, it would be easy enough to find.

[gilgongo]

The genre-based system is not ideal and has some clear disadvantages. So if the CS limit issue could be solved in a different way that would be nice.

so while the lowest ping may not appear first, it would be easy enough to find.

That's an an interesting observation I think because most people will only have a handful of fast enough servers to use anyway, and once they find one they're probably not going to care about sort order after that. So a "rough" sort on location (however that's done) should be fine even in a list of 50 I would think. Similarly with a fresh button: servers and players don't churn very fast, so I doubt you'd need to hit the button more than a couple of times before joining a server for a jam.

I take it also that servers would attempt to register on all central servers and go with the first free slot they were given?

[gustebeast]

with a fresh button: servers and players don't churn very fast

It may be advantageous to separate the query that gets the 50 servers from the query that checks players. With the current system, does each server report the players to the central server, then the central server reports the players to the client, or does it work like ping where each client asks the server directly?

I take it also that servers would attempt to register on all central servers and go with the first free slot they were given?

What do you mean by a "free slot"? The simplest set up would be to have one central server with a single unified list of servers. Depending on the load it gets, this could be done with a single beefy computer or require load balancing between multiple computers (still with a single domain and shared unified list). Load balancing may be tricky, although we could tackle it piece by piece, start by implementing the distance sorting, then unify the genres.

We need a way to avoid the inevitable issue of a genre having too many servers in it (unless we want to have a jazz1 and jazz2). However, I think this proposal does more than that: it paves the way for some nice improvements to the user experience:

1. User opens Jamulus for the first time, opens server list.
2. User is quickly presented with the 50 closest servers. The server name, location and genre columns appear right away, the ping and musicians populate after a second or two (and continue to update every couple seconds).
3. The user opens up the "Genre" filter dropdown and selects "Jazz" and "All Genres". The list then updates to show the 50 closest servers that have their genre set to "Jazz" or "All Genres".

If the user opens Jamulus back up, it would remember the filters so the servers would appear in the same order right away. We could still support filtering by server name or occupied, the server would just find the 50 closest servers that match the name/have musicians.

[gene96817]

Please don't implement a "nearest 50 server" list. I recently moved across 5 timezones. I now have Jamulus friends across 8 timezones. I won't see my friends from my old place with only the nearest 50 servers displayed. Also, my old friends won't see my new server and visit. It has been really great to have old friends dropping in, even if the total delay is >140ms.

[gene96817]

I just checked Jamulus Explorer and there are 78 servers under Genre Classical/Folk/Choir. My client shows 38. Any clue why the number of servers are different?

[ghost]

Maybe Jamulus should display all servers (ip addrs Or fqdn Or whatever can be displayed) in the list before pinging them instead of not displaying the unpinged servers. Then the list can be sorted by the client as it pings (the way it is now). I think adding heuristics etc. is just adding bloat.
Edit:
I just found another Undocumented Jamulus feature that solves this problem:
Jamulus --showallservers
please add this documentation to Jamulus --help AND the documentation page.

[gustebeast]

Would these options be sufficient for your use case?

1. At the bottom of the connection window, there would be a "Next Page" button so you can see servers that are further than the closest 50.
2. We could add the ability to "favorite" a server so you could quickly see those even if they span large distances.
3. You could filter out empty servers to keep the list under 50 (provided you are looking for online friends).

Additionally, we can make the page size larger than 50. Could probably make it the current max server count per genre without hurting performance.

[gustebeast]

The problem with sorting by ping is that you need to ping every server in the list. Creating separate lists per genre has worked for now, but that isn't a long term solution. We shouldn't need to separate "default" and "All Genres", and it should be possible to see all genres in one list if you don't care about what genre you want to play/listen to. This is made especially painful because it takes a while for the servers to sort when switching genres.

The location heuristic I think has the potential to actually remove bloat both in the user experience and in the code. For example, we could get rid of the logic that prevents ping sort order changes on mouse hover.

[gene96817]

@gustebeast I would prefer the connection window scroll. "Next" would work also work. It would be nice for the connection window to indicate the number of servers detected. Keep the list simple and just sort by ping times.

[gene96817]

If you don't sort the server list by ping, how will I know how long it will take my packets to reach the server? Also, the ping times could change by network traffic (e.g. time of day). Of course, we don't want the list to churn because of jitter in the ping times. I am sensitive to this issue because I don't live in the center of a metropolitan city. Locations at the edge of the city or further see more delay from fluctuations in network trafic.

[gustebeast]

The ping would still be displayed in the same place. It's just a matter of how the list is sorted. If distance is a good enough approximation of ping, you'll still see the low ping servers at the top of the list. Per pljones' example above, there will be cases where a server with a lower ping appears lower in the list, but so long as it is still close to the top, it should be easy enough to look at the pings and pick out the best one.

[gene96817]

What if the client keeps a single list of all servers, only the selected subset would be displayed. I agree that we don't want to be pinging all the servers frequently. We could ping the serves in a slow background process. If we keep the list between sessions, the old data should be a good starting approximation. Then only new users starting the client for the first time would need to have all the servers pinged.

I often see some servers that are farther away with shorter ping times. It depends on whether those servers are close to a major backbone.

[gilgongo]

Hi all - this seems to be more of a general discussion about central servers. Until we have some agreed definable action to take, I'll move it to a discussion (and update the title) if that's OK.

[dingodoppelt]

What I found to be useful is to just make the serverlist sortable and make it sort by most musicians on a server per default.
I personally don't care for ping times and find it awkward to have the serverlist sort by ping time in the hardcoded way that it is now (including the workaround with moving your mousecursor off the list). You don't have to scroll too far down the list to see everybody playing on a given serverlist. I just rarely feel the need to sort by anything other than most musicians on a server, but in case I do I'm able to click on "Ping time" or "Location" to do so.
I took on the problem in my fork if anybody wants to give it a try -> https://github.com/dingodoppelt/jamulus/tree/sortableserverlist
And no, --showallservers doesn't do the job since it prefixes the servernames with a number and interprets the number of musicians as a string though it lets you sort the list by clicking the tabs.

[pljones]

I use both constantly... (changing them depending what I'm after)

[ghost]

Note that all the columns are sortable with --showallservers. Maybe it is not so hard to make all the columns sortable without --showallservers. (FYI I'm not suggesting --showallservers should be the default anymore)

[dingodoppelt]

@DavidSavinkoff
as I wrote here #1035 (comment)

I took on the problem in my fork if anybody wants to give it a try -> https://github.com/dingodoppelt/jamulus/tree/sortableserverlist
And no, --showallservers doesn't do the job since it prefixes the servernames with a number and interprets the number of musicians as a string though it lets you sort the list by clicking the tabs.

Maybe it is not so hard to make all the columns sortable without --showallservers.

It isn't hard at all, this is why I changed it in my fork ;)

[gustebeast]

If we got rid of ping sorting completely, that would solve the "pinging too many servers" problem and allow us to display all the servers in a single list with pagination and genre filtering. However, I think people will always want the option to sort by ping, if for example they have their own friends joining and just want the best connection.

For me the main question here is, how can we combine all the genres into a single paginated server list and still support sorting by ping. My proposal was to sort by distance first, then compute ping for the closest 50 servers. The lowest ping won't necessarily appear at the top of the list, but it'll surely appear on the first page, and generally lower pings should appear higher. I'm not sure how difficult it would be to sort by distance. Maybe the central server could compute the location of registered servers using their IP, then send those locations to clients when the connection window is opened.

[gene96817]

In the past, I have proposed a possible low ping rate approached to managing the server list.

Propagation distance (latency) is essential for my community because most servers are geographically far away (> 5000 km). Visitors come when they feel they are not "too far" away.

IP addresses are not tightly correlated to geographic location. Also, ping times are not tightly correlated to geographic distance. The Internet router paths between two points can be significantly longer than linear distance.

I also believe a single list can improve the management of genres, especially groups that enjoy music from more than one genre.

[pljones]

My proposal was to sort by distance first,

We measure distance by ping, though.

[gustebeast]

We have some disagreements, but hopefully we can agree that having Any Genre 1-3 isn't a good solution, especially for new people coming to the platform.

I proposed sorting by distance as a means of querying a larger list of servers without spamming too many pings. @gene96817 made good points about there being people for whom that heuristic won't work well (I still have a feeling it would work well enough to get the server you want to join in the top 20). What if we sort by distance by default, then allow the user to sort by ping if desired. When they sort by ping, the client pings all the servers (in a round robin if there are a lot of them). This goes on for a minute or so, then repeats at a slow interval.

We can reduce the issue of too much pinging if we make distance the default, and employee some techniques to limit the number of pings for those who choose to sort by ping.

Also, I feel like we haven't discussed the other big element here:

At the moment, the list can get fragmented if too large and, where UDP fragmentation causes routing issues, this can cause the list to become only partially visible.

Are there any leads on how to fix this? Seems like it should be strait forward to send a block of data down to a client, although I'm used to web development where that stuff is easy.

[gilgongo]

@gustebeast Can you clarify what you mean by "distance" if it's not ping?

Also just to be clear, the reason why we have the "genre system" is that it was the best of a bad choice of strategies to overcome the fact that each directory can only list a maximum of 150 servers. Any "combined list" solution would have to account for this.

As to the UDP issue, the discussion on that is here #1058

[dcorson-ticino-com]

He means physical location distance by some magical algorithm.
We have about 20% of the servers in the Genre 1 without location info, for example.
Using IP is also not really helpful. Take my IP and you get a location which is 300km from where I really am.
Ping is the only useful indication of distance for our use here IMHO.

We presently can support just over 1000 servers. If we put all that on one server we will initiate a tremendous amount of ping traffic every time the connect window is opened. I see no good way to treat this off of hand.
I had started to implement favorites for the connect window a while ago. I think that would be the simplest way to handle this. Add a button that puts the server that one is presently connected to in the favorites list. This would also work for private servers.
I will look at this again, but am not sure that it is on the top of our priority list right now.

[gilgongo]

@dcorson-ticino-com Ah OK. That's what I thought.

We presently can support just over 1000 servers. If we put all that on one server we will initiate a tremendous amount of ping

My own opinion on all this is that we should approach that problem from a different point of view:

I think the following statements are true:

1. There are too many empty servers. Empty servers discourage people jamming/playing on them (note that once a server has more than two players on it, numbers tend to shoot up).
2. To an extent, ping time is a function of physical distance.
3. Usually, it only makes sense to connect to a server with a low ping time.

Therefore, I would say limit servers to a maximum number by either location (so server operators would need to choose from a fixed list of cities) or idle time (so the directory servers would de-register them after a few hours without players).

Doing so would I think result in:

A. Increased numbers of people playing together (due to point 1)
B. A solution to the whole issue of the 150 server limit, ping traffic and unstable UDP packets.
C: (Possibly) allowing people in disadvantaged time zones relative to the US and Europe to have a fair chance of registering their servers.

[dcorson-ticino-com]

I think that the server use paradigm is very different for the groups 1) we come together and jam, and 2) we belong to a group and schedule a rehearsal.
What you are saying Jonathan presumes, correct me if I am wrong, group 1).
I think the people from group 2) want to have a fixed base where they can connect and work, as they would go to a rehearsal room once or twice a week. That is, at least, the reason why my server is on all the time. In the mean time it is used by several groups which see it as their rehearsal room. If they do not find it they will not go somewhere else, they will stop and call me. But they use it only a couple of hours a week, so it appears to be unused. If the server was logged out after a couple of hours it would mean that I need to be available and near a computer all the time when they request to use the server to start it again so it registers.

I do not have a real solution, but think that if we implement favorites which also include private servers more people would be willing to use private servers as it would still be easy for the clients to connect, without typing IP addresses in more than once. I know this is possible now too, but favorites might make this way forward more palatable.

(Note that I think that grouping servers by physical location realistically is impossible as the physical location correlates only weakly to the ping time. How should the algorithm know that I have better ping times to most servers in Italy, Germany and France than any other server in Switzerland?)

[gilgongo]

What you are saying Jonathan presumes, correct me if I am wrong, group 1).

Yes. As to Group 2, I agree your use case is real, but fundamentally that use case has always meant to have been handled by private servers. The "jam" in the name "Jamulus" is (sort of) there for that reason.

But to circle back slightly: do you or do you not think that empty servers discourage people jamming/playing on them? If you don't, then efforts to constantly increase server numbers make sense. If you do, then it doesn't make sense to try to have more servers on line.

[pljones]

If we had geographically distributed directory servers (which we don't -- mine and the Franfurt ones aren't far enough apart in ping time), then a registering server that wanted to provide a public service for jamming (and that is a distinction that needs emphasising) could choose the nearest directory with space to register by default.

Similarly, a client run for the first time could ping each directory server a few times to get their ping time and then default to the fastest. That might provide the user with a better experience.

Neither of the above should need protocol changes, as far as I can see - they're just different strategies over the existing ping message.

But it does need investment in the infrastructure.

[gilgongo]

@pljones OK, but I'd ask you the same question as I did @dcorson-ticino-com: do you think that empty servers discourage people jamming/playing on them? If you don't, then efforts to constantly increase server numbers make sense. If you do, then it doesn't make sense to try to have more servers on line. It makes sense to have less.

[dcorson-ticino-com]

(Replying here to get out of the "View more" trap. Is there any way to turn that off so all is always shown? BTW anyone know what this "1" with up and down arrow means at the left of all the posts? I can't find anything in the GitHub doc and information theory would tell us we can remove anything that is always the same without loosing any info.)

For my user group, people look for empty servers with the lowest ping possible to use. For example when their customary server is having a bad day. It has happened to us that one person had a very bad ping one day so we used another server where everyone was OK.

After saying above that geographical sorting will not work, I wonder what a geographical sort over all servers within a 700km radius would result in. 700km radius would, for me, include almost all of the good servers, but would it reduce the server list enough to be a reasonable basis for starting to ping? If we went this direction we would have to require geographical information from all servers, which we now don't have from about 20%, and also a way of entering the position of the client.

First step, I think, would be to find someone with experience with geo SW to make a quick hack to see if it would be even worth it to try out going in this direction. I guess using info from the Jamulus explorer this could be done offline. If this doesn't reduce the list of possible servers enough to be realistic we will need to find another idea.

[gilgongo]

(Yes, I was wondering about that "view more" and the "1" business. Mysterious.)

First step, I think, would be to find someone with experience with geo SW

I'd worry we'd be at the mercy of commercial providers for GeoIP data though. MaxMind's are free for basic stuff for moment (I use their lists for blocking traffic outside of the UK for a project I've got), but they recently went more restrictive I think. Better to have the server owners declare the city they are in I would think (even though that's not trivial either). And in both cases we'd need to calculate the distances, which seems... difficult, but then I'm not an engineer.

Still curious about what you think of my assertion about empty servers though. I hear what you are saying about your own circumstances, but "canonically" if you like, that use case is for private servers.

[pljones]

An empty server explicitly means one thing only: no one is using that server.

There are a number of possible state transitions:

• (to be pedantic, this isn't a transition but) nothing happens, it remains empty
• one person joins as they've had no luck elsewhere and the server provides a decent ping, hoping other will turn up for a jam
• a band turns up hoping no one else will turn up as they've managed to get a server with a decent ping for all of them

(OK, the last case could be broken into the individual client connections but that's missing the point.)

(My point being, you can't get some random bunch of people turning up without one of them being first.)

From there, there are two new states: occupied by a solitary person and occupied by a band.

@gilgongo, if I understand correctly, you're suggesting the server with the band hoping for an uninterrupted practice session is likely to attract more people than the solitary player?

Personally, I'm put off joining any session with a drummer, as I'm a drummer. That often ends up with me being in that "solitary player" state. I find people drop in just about as often as new people join a jam with multiple players - if I'm keeping track, which is of course harder.

[gilgongo]

you're suggesting the server with the band hoping for an uninterrupted practice session is likely to attract more people than the solitary player?

No I think that confuses two unrelated points. The first point is that if you see a server with one or more people in it, then you are much more likely to join than if it has none. By extension, I'm saying that if you have fewer servers available then that increases the chances of somebody crossing that barrier and entering alone, which in turn very much increases the chances of more people joining after that.

The second point is about an existing group of people who wish not to jam with random strangers. For better or worse, I'm saying those people should set up a private server (for now at least). It's a completely separate point I think.

I find people drop in just about as often as new people join a jam with multiple players

I'm saying empty is a lot less likely to get people in than one person only though.

[dcorson-ticino-com]

It may well be that a used server attracts people.
We have some guests at almost every rehearsal who stay and listen for varying amounts of time.
Only once did we have a "disturber", but we quickly muted and then they left.
In the mean time we don't pay any attention to what is happening on the server, while playing our attention is elsewhere.

[pljones]

The first point is that if you see a server with one or more people in it, then you are much more likely to join than if it has none. By extension, I'm saying that if you have fewer servers available then that increases the chances of somebody crossing that barrier and entering alone, which in turn very much increases the chances of more people joining after that.

But you haven't explained how you achieve the first state -- or how having more occupied servers means more people will join an unoccupied server... In fact, you seem to be saying the opposite.

I'd agree that anyone wanting to have a jam is likely to look for somewhere that isn't unoccupied. But I don't think having unoccupied servers stops people joining unoccupied servers. Or puts them off joining occupied ones. So I don't actually see any connection.

[gilgongo]

@dcorson-ticino-com

It may well be that a used server attracts people.

It would seem to stand to reason at least.

@pljones

But you haven't explained how you achieve the first state

I don't mean imply that having more occupied servers means more people will join an unoccupied server (at least, I don't think that stands to reason). Rather, having just a few servers (say three to five per city) raises the probability that people will join them. Put it this way: if there was only one server, then everyone would be in it :-)

This is just an hypothesis of course. But it seems worth perusing because we can observe that increasing the number of servers does nothing for occupancy rates - and we have this 150 server limit and associated problems when we do.

BTW, you said earlier:

If we had geographically distributed directory servers ... then a registering server that wanted to provide a public service for jamming ... could choose the nearest directory with space to register by default.

Could we do that by geofencing the directories? So instead of the genre system we have servers that only accept connections from certain regions?

[pljones]

No. No fencing. Just pick the "lowest ping" when registering a server in a directory or choosing the default list. If you're in South Korea and the nearest directory is San Francisco, that's fine. Or maybe there's one in Tokyo with a lower ping, even better. Or maybe there's nothing with a lower ping than Frankfurt still. It doesn't matter - pick the lowest ping.

It needs a few pings to each directory on the list, like I say, to get a meaningful average time to do the sort, but still not too long if done once at program start up.

[pljones]

I had another idea :)

For the existing directory servers, change the names (again...) to all have "public" at the front. Both client and server UI dropdowns would have these listed. These would be intended for any server provider granting public access to their server.

In addition, in the server UI dropdown (but not the client), add two directory servers with "private" at the front. These would be for server providers who did not intend them to be generally known (so Jamulus Explorer, as a client, should not show them) but to help with the ping-back issue. Ideally server operators would provide their hostname to their userbase, rather than instructing them to enter the private directory server into the client UI. (I'd hesitate before saying prevent that happening, though.)

Currently Volker provides one "official" private directory server (I forget the name) and I've got a bunch on the "unused" ports where the genre-specific servers currently aren't -- 22124, 22224, 22624 and 22824.

[pljones]

Any registered server choosing a "public" directory server would be intended for use by the public. Any registered server choosing a "private" directory server would not be intended for use by the public -- but there would be no restriction implemented to prevent this. It's only that the directory list is a little less public.

[dcorson-ticino-com]

I had another idea :)

Good !

Another question; as I understand it private servers don't have a name or location. Is there a reason for that?
If we have favorites that include private servers it would be good if they could have names, etc. too.
Would it be possible (or is it already possible) to use --serverinfo and have a way to send that to the clients.
D

[gilgongo]

Sure - I was merely wondering about the terminology, which would clash with what we had now (since "private server" would mean two things). Not a big problem.

[gilgongo]

If we have favorites that include private servers it would be good if they could have names, etc. too.

The vocab confusion is already starting! :-) When you say "private servers" I take it you don't mean what @pljones is describing, since they would have names.

[dcorson-ticino-com]

You said it.
I am talking about the OLD private servers that are not taking part in any directory/central server publicity.

[gustebeast]

As to the UDP issue, the discussion on that is here #1058

The 150 server limit may actually be independent from the client-side problem described here. If we can figure out how to display a large number of servers in a unified list without over-pinging, clients could query all the central servers and compile them into a single list.

Responding to the concern about empty servers, private servers, etc. let me summarize a few user stories:

1. I want to jam with other people. I will either join an empty or non-empty server based on ping times/genre/who's playing.
2. I want to jam with my friends, but happy if others join. I will join an empty server with the best connection to my group.
3. I want to jam only with my friends. I will create a private server, or join a public server with the best connection to my group and "solo" them.

It seems like #1 could be addressed by toggling on/off the "hide empty servers" filter. Maybe it would help if there was a way to highlight/promote the non-empty servers without hiding the empty ones.

Having a large number of servers is good for #2 since it gives you more options for finding a closest/best ping server.

Do I need to host GTB Jam? Probably not, but it may provide better ping to someone, and maybe CityOfTinyLights won't be online 24/7.

For #3 I feel similarly to @pljones. We allow private servers to appear in the connection window, but behind a filter that defaults to public. We would either need a way to opt out and fully hide your server, or add a server password for those who want extra assurance no one join.

There are a number of techniques being discussed to reduce the number of servers we need to show to the user. This is good both for reducing pings and helping the user get connected more easily. This may be enough to get Any Genre 1-3 down to just Any Genre, but I doubt it will be enough to merge all the genres into a single list. To fully solve the scalability issue, we need a way to sort a list of servers by ping without regularly pinging every server.

We may be able to do this with a clever pinging strategy, but I still think distance is our best bet. It has two major flaws:

1. Doesn't correlate fully with ping times.
2. Can't be reliably determined by IP, requires input from users and server operators.

To address these, here's my proposal:
We introduce a new distance filter:

• 150 closest (default)
• 150-300 closest
• ...
• No location 1
• No location 2
• ...

Once the filter is selected, we can sort the servers by ping. The best ping server may not be the closest, but presumably it is at least in the closest 150. That should mitigate flaw 1.

Many servers already provide their location, but it is just a user provided string. Some are quite vague, some use abbreviations like "CDMX, Mexico". I propose that we replace the current -o city string + country code with a single identifier for your server's city. That will allow us to get accurate geocoordinates and more consistent location names for the connection window. For servers still using the city name, we can make a one-time hardcoded mapping of names to identifiers (i.e. "CMDX, Mexico" => "MX-MEX"). For those that don't have any location, we put them in the "no location" buckets mentioned above, sorted alphabetically perhaps.

For the client, if you try to open the connection window without a location, we would need to show a dialog asking for it, ideally prefilling with a best guess. We could store this location in your profile, but we would need to clarify that it is private. Even better would be if you could choose to make it public.

Once we have the location identifiers, we just need to map them into geocoordinates (Spreadsheet from World Cities Database), and then we can compute distance.

[gene96817]

How accurate would you want the geolocation? How should we handle the smaller cities or suburban/rural towns? I observe that suburban areas have higher latency/Km than major population centers. (And transocean paths have lower latency/Km.)

[gilgongo]

I want to jam with other people. I will either join an empty or non-empty server based on ping times/genre/who's playing.

I'd modify that to be:

"I want to jam with other people. I will not join an empty server if I see a non-empty server based on ping times/genre/who's playing."

People don't want to join empty rooms. Ergo, reduce the number of servers, lower the "ghost town effect". If you fire up Jamulus and see 100 servers, two of which are full and all the rest empty, who's to blame you from disconnecting and watching Netflix instead?

For all we know, the public servers on Jamulus could be way more active than they are.

[gustebeast]

How accurate would you want the geolocation?

By accurate do you mean, how many cities should be included in the spreadsheet? If your city isn't included you'll have to pick a neighboring one, which would make it less accurate. The world cities database has a larger option that requires a one time payment, I'd be happy to contribute for that if need be, but maybe we could just let people enter their coordinates manually to handle edge cases.

I will not join an empty server if I see a non-empty server

There are a number of reasons to choose an empty server and wait for others rather than go with a non-empty one: ping time, genre, instruments, number of people, etc. I agree with you though about the ghost town effect, I think especially for new users we should find default filters that highlight occupied servers.

[gene96817]

I still think the jumbo UDP problems can be fixed by having Jamulus breakdown the payload to send only small UDP packets. This would then require sending the table in N fragments where each fragment is labeled M-of-N. A lost fragment can be individually requested by the client.

The only real distance we have is ping time. Trying to geolocate IP addresses will yield wildly inaccurate positions. Since many servers are up for long periods of time, a slow ping update is adequate. Only a newly installed server will wish for a quick ping update.

If the client maintains a persistent server list, it creates opportunities to reduce ping traffic. Consider (1) the client could ask for only servers that had a status change since time T, when the client was last running. (2) most of the server ping times would be similar to the last session. We could immediately update the servers that had less than 70 ms ping, and then do a slow ping update on everything else. (3) the only time the user would notice bad server entry is when the server went offline and no longer answers to a ping. I believe only a restructuring of how we manage ping updates (similar to this) is necessary to significantly reduce the global ping traffic.

[ghost]

How to make list shorter:

• Central server reads your ping value and returns results that are +- 30 milliseconds from this value.
• Think about the possibilities.
• Geolocation not required because it is addressed as a non-trivial function of milliseconds.

[gilgongo]

Thus client wants a server list that contains servers (300 +/- 30)ms away from the London ping time radius.

But why would I want that? I want servers with ping times of 10-20ms, not 270-330ms. I'm not playing on the central server, am I?

You say "Think about the possibilities". Can you name some of those? Maybe that might help me understand.

[dcorson-ticino-com]

Then a server in Montevideo or Santiago de Chile or Windhoek or Mumbai or ... that also may have 300 +/- 30ms to the central server would also be seen as being close to Tokyo ?

[gene96817]

We are getting close with the Tokyo example. Consider the Central Server is approximately at 0º latitude. Now the client is pinging from 270º. How will the Central Server know not to serve up a list of servers from 90º? The earth is a globe. We have a trigonometry problem. The pings, as just radial distance, is not enough to drive the server selection.

[ghost]

The point is to get a significantly Shortened list that is Guaranteed to have the results you want. This is instead of giving you a full (brute force) list as is done currently. This method does not prevent you from combining it with other methods.

Think of a sheet of paper 300x300 mm square.
Note that there is no data outside of the 300x300mm area.

Draw two concentric circles at radius = 150mm and 90mm (120+/-30)mm
Area of 300x300mm =90,000 mm^2
Area between circles = 45,239 mm^2
List is half as long and contains everything you need.

[gene96817]

The point is to get a significantly Shortened list that is Guaranteed to have the results you want.

Give this alternative a serious read. It promises to provide an even shorter update list.
-- Consider a client that saves the server list between sessions. The list of servers seems to have a very low chun. On startup, the client request a list update from the Central Server for all servers that have started/restarted since the last time the client's server list was updated.
-- The Central Server sends a list of only "recently" started servers for the client. For frequent users, this would be a very short list. For a new user, this could be the complete list.
-- The updated list has every server the user could want. Because of the low server churn, a short update list is all that is needed.
-- The need for pings does not change. However, note that I don't see most pings changing very much. Only servers that are new to the client list need an immediate ping measurement.

[gustebeast]

So we have three proposals for solving the client-side server count scalability issue:

1. Have central server calculate ping difference between client-central and server-central.
2. Cache ping results and avoid re-pinging servers that have high ping.
3. Ask server operators and clients to give us a location/city.

Number 1 is quite intriguing, but you'll use up your 150 server budget on far away servers rather than pulling ones that are closer (due to it pulling from a circle around the central server). It still may be good enough to get your desired servers in the top 150, but I'm somewhat doubtful. It will also require more pings to the central server to establish an accurate client-central ping.

Number 2 is promising, and can be implemented without any changes from server operators or new profile information from clients. I am hesitant though because this hinges on our ability to hide the ping reduction logic. I think it is bound to lead to confusion around why certain servers get ping updates and others do not, although with some time and polish we could address each confusion and improve our heuristics.

I like number 3 in part because it is a fairly simple/intuitive design. You specify your location, then you can select 0-150 closest, 150-300 closest, etc. You'll get consistent results each time, and aren't left wondering why a server appears where it does. The additional friction to server operators and clients is a tough pill, but it is also a one time deal.

[gilgongo]

Yes, C is addressing a separate (but real) issue so isn't very comparable. And as you point out, maybe easier to do with just as label, unless the behaviour of "open" vs "closed" is materially different (eg the former expire on empty).

As far as I can see, the "150 limit", and its attendant UDP fragmentation issue, can only be solved in the current architecture by some form of list truncation. I'd favour a combination of B and C (since C also addresses the "can I join this server?" issue that seems to be a reported problem). At least that seems fairly simple. Getting clients to give a location/city might be worth looking at, but seems more complicated and would rely on something like MaxMind who might start charging for their data.

EDIT: The other alternative is to implement a TCP layer into the protocol in some way. That might not only address the problems under discussion, but also a bunch of other things perhaps.

[dcorson-ticino-com]

B) has no chance of working. If London is full, just use Clerkenwell or any other place name in or near London. Why not Manchester? As we know, even Frankfurt is ping-wise close to London.

For C) we could go a geographic route, Closed Servers Europe, Closed Servers NA West, Closed Servers NA East, etc. and try to have the directories local to reduce response times.

For info, I took the total list of servers online yesterday from the Jamulus Explorer and sorted by country. Using simply the countries limitrophe to Switzerland reduced the list from almost 700 to 190. If that filtering was done properly by distance, say 500km radius, the list would be reduced much more. So this looks to me like it could work as a filtering principal. The problem then being getting the true position of the servers and clients. I see that even the GeoIP providers don't claim accuracy of better than 50 to 70% to have less than 200km error. That is not really helpful.

[gilgongo]

B) has no chance of working.

Define "working" in that case. There are a number of different problems to solve. I;m assuming the list of cities would be fixed BTW, so Clerkenwell would't be there.

As we've established I think, C also potentially reduces the numbers, but right now we see some directories at or very near full, so would they not all just fill up again and we'd back to the start (only with even more of a "ghost town" vicious cycle)?

The problem then being getting the true position of the servers and clients.

Why not just use the existing ping mechanism? It's not a problem if it's only on a small number.

[pljones]

Naming the directory servers is totally different to having some complex "timeout" eviction system that may never allow a server to be visible when it needs to be without totally rewriting the registration approach.

As we've established I think, C also potentially reduces the numbers, but right now we see some directories at or very near full, so would they not all just fill up again and we'd back to the start (only with even more of a "ghost town" vicious cycle)?

Some are full. Some are empty. Are you trying to force people to balance the CPU capacity the rent from cloud providers so there's one auto-magically self-partitioning server that everyone join and put their own walls up in? I'm not following now at all...

---

We've got a list of directory server names that can point via DNS anywhere in the world. All we need are:

• drop the genre lists (no protocol change)
• directory servers more geographically distributed (no protocol change)
• "default" choice of directory based on lowest average ping with space available (no protocol change)
• maybe add to the server N "private" or "closed" directory servers (no protocol change)

[gilgongo]

Naming the directory servers is totally different to having some complex "timeout" eviction system

I'm not advocating timeouts.

Some are full. Some are empty.

I'm talking about directories, not servers. This is all getting too confused.

Your points address problems that are different from the problems that I'm talking about. We already have enough public servers, but the vast majority are empty. Trying to raise the numbers still further isn't a problem that I think needs solving.

I agree that your last point is a good one though because there does seem to be a real issue with people wondering whether a server is actually public or not. Although hard to know why server operators don't generally state this welcome messages - perhaps they don't realise.

[passing]

Hi Guys,

I am having a few thoughts on the topic and hope I can help with these:

Jamulus uses UDP to be able to do Audio streaming with as little latency as possible.
From what I understood, the Central Server functionality was added later as there was high demand and a quick and easy solution was wanted (which did a very good job so far but now has reached certain limitations). But, in my opinion, there is no need to use UDP for that at all: There's just no need to have low latency to exchange server lists and then deal with lost packets and a limit of 150 Servers.
Also I guess, the central server component does not actually need to run on a server that is a Jamulus (Audio) Server itself (?).
Logically, the central Server could be a separate component/project: No need to be tuned to low latency. You might even write it in another language, give it a well defined API, unit tests, etc.

Regarding the idea to use Pings to estimate distance, I see the following problem:

• If i am located in Frankfurt but my Internet connection is bad (e.g. using Wifi or someone in my household is just making a phone call which has priority), my ping to the central Server may indicate a higher distance than I have. So I would get Servers that are further away from Frankfurt rather the ones that have the shortest ping from me.
• Other scenarios could be like this (not based on facts): Someone in Ireland may have quite good pings to servers in the UK (while there are not many servers in Ireland). The central server would return servers with a ping similar to Ireland-Frankfurt - maybe in Norway or other locations far away from Ireland, but would not return servers in the UK which would be the ones with the shortest pings from the client.

I'd suggest to focus on filters in the server lists, which the client has full control of, rather than trying to guess, which servers are best for him. Filtering could support:

• country (possibly with multi-select)
• genre
• include/exclude of: completely empty, with connections, completely full
• clients favourites, and maybe even based on ratings (how good was your connection?)

[gene96817]

One side effect of pings by the Central Server is that the pings allow the Central Server to detect servers that are down and should be removed from the list.

[passing]

The client requests one of the server lists from the directory server.

@dcorson-ticino-com I don't think that's the case, but each server list is actually served by a different server / instance:
https://github.com/jamulussoftware/jamulus/blob/master/src/global.h#L99-L107
I don't see a good reason for keeping this 'architecture' and it has several disadvantages, e.g.:

• Whenever there is the need to add a new server list (e.g. for another genre, or when all server lists are full) it is necessary to start another central server instance, give it another DNS entry, and finally to release a new jamulus version that has that new server list hardcoded.
• It makes it hard to implement proper filtering as none of the central server instances knows about all servers that are registered
• It requires that server operators update their configuration whenever they lost a slot on the central server instance which is completely full

From an architectural point of view, I can just recommend to not create a solution that is limited to any specific requirement of reducing the number of servers shown to the user:

• Better build a solution that is technically able to keep track of and serve a big number of servers.
• Then limit that solution by configuration where needed:
  • max. numbers of servers that are returned to the client
  • min. server version required to be listed
  • max. numbers of pings to be made from the client
  • ...
• Then you are flexible to implement different filtering options in the client (or on the server), which could also be a "smart filter" that tries to identify the servers that will be best for the client before the client has pinged them

[passing]

One side effect of pings by the Central Server is that the pings allow the Central Server to detect servers that are down and should be removed from the list.

servers that are down for 33 minutes get removed anyway:
https://github.com/jamulussoftware/jamulus/blob/master/src/global.h#L221
not sure if it is necessary to shorten this?!

[gene96817]

// time-out until a registered server is deleted from the server list if no
// new registering was made in minutes
#define SERVLIST_TIME_OUT_MINUTES 33 // minutes (should include 3 UDP registration messages)

Does this say a server continues to send "new registering" messages?
I was not suggesting the time-out to be shortened. You don't want a temporary network congestion to cause a server to be declared down.

[passing]

@gene96817
I just meant that one might want a server to be removed faster when it goes down.
And yes, servers re-register every 15 minutes, that means they won't be removed as long as that re-registration doesn't fail more than twice in a row

[chrisrimple]

I'm a late participant to the discussion, but it seems to me that the core problem is (1) central servers are limited to 150 listed servers and (2) there aren't enough central servers for all the servers that want to be listed. What about (A) allowing a central server to redirect a query to one of its listed servers and (B) having all servers be able to respond? Go a step further and have the central server redirect ALL queries to listed servers, essentially serving as a load balancer for the queries.

[chrisrimple]

Doesn't a "show only in-use servers" checkbox, perhaps checked on after install, solve the "ghost town" problem regardless of how many public servers are running?

Rather than solving in that way, what about allowing a server to publish "open jam times" back to the central server, and consolidate those posts to a view in the client? That would allow a new Jamulus user seeing a "ghost town" to at least know when to come back online to join a jam.

[pljones]

What about (A) allowing a central server to redirect a query to one of its listed servers and (B) having all servers be able to respond?

Any solution supplying a larger response to the client means more clients are going to experience UDP fragmentation and become more likely to fail to get details for servers "near" (low ping) to them.

So that's blocked until the UDP fragmentation issue is overcome.

Unifying the list at the client (i.e. having the client query all directory lists) means greater ping traffic for every server (significantly greater - about three times and growing fast, just based on the number of servers, let alone the number of clients).

Increasing the size of the individual server entries in the response from the directory servers -- by adding more information -- both causes increasing fragmentation and higher traffic, of course. (The current solution for publishing availability schedules would be "put that in your welcome message", I guess.)

[gilgongo]

It seems I may be the only person bothered by the extremely poor occupancy rate of public servers, but if you could indulge me for a second:

If we found some method of increasing the number of servers available beyond the 150 limit, and we had, say 1,500 servers on each directory - what would that achieve beyond getting rid of genres? Huge numbers would still be empty (even more than now unless player numbers also rose) and the numbers of playable servers per person would presumably be largely unaltered.

Should we not therefore be more interested in ways of making Jamulus look more inviting with fast, reliable and above all occupied servers instead? This is a totally separate issue to having "private" public servers, which I grant is a real issue too.

Server number reduction on its own isn't an answer perhaps, but multiplication seems not be up to much either.

[pljones]

It depends on the view of "inviting", I guess. The client has an existing feature that allows you to hide empty servers, so it's not like people have to be put off by them. If they want an empty server, they can find plenty. If not, filter them out.

[gilgongo]

If they want an empty server, they can find plenty. If not, filter them out.

I don't see how that's a solution to what I'm saying though. Similarly with @chrisrimple - I have no problem with sorting or clicking or making things checkboxen with scrollable columns, etc. Those things may solve some small problems for some people in niche circumstances where they're not interested in ping times or something, but I'm talking about a bigger issue.

But like I said, I seem to be the only person worried about the effect of having ever-growing numbers of largely empty servers. So I'll have to accept that.

[chrisrimple]

@gilgongo I've previously requested that we make the column headers clickable on the Connect Setup window, so a user can sort by Server Name, Ping, etc. Maybe add a column for # Of Participants, then make it the default sort? Or add a checkbox for "Show Only Servers In Use" and have it be checked on by default? Or even break the list in two, with separate scrollable tables for In Use and Not In Use servers? There are lots of ways to better expose the In Use servers, we just have to pick one. :)

[gilgongo]

This is really a separate issue I think - the problem in my opinion isn't about the exposure of in-use servers in the UI. People have no problem spotting those.

[ghost]

Here is how to refine 'ping radius':
Use more than one directory server, for example 3 Rock Genre (fail-over even) servers in different locations that share their lists and pings.
This makes it possible for any 'one' of these servers to calculate 3 ping radii and send the intersection of the 3 sets.
This will substantially reduce the list sent to the client.

Of course the client will have to supply the 3 servers with appropriate ping radii, either by responding or initiating pings.

[ghost]

Ping time is a single variable, thus, you can Only specify time elapsed (there is no direction). If you want two variables, you will need two directory servers that share lists (and you will need to ping them both to get two variables).
A few strategically placed directory servers will give increasingly better results.

[gene96817]

Ping is a scalar. It is the only variable that affects the delay of a jam session. My point is that for a given user, the central server cannot know which servers are best for a user. Yes, you can use two servers to measure ping times but that work is unnecessary. (And in 3-D space, you need three ping points.) Just let the clients do that work.

[ghost]

If every jamulus client pings every jamulus server every few minutes (the current situation), then at some point there is more pinging O(n^2) than jamming. Thus, the directory must do something to stop the clients from pinging so many servers. The methods I'm proposing reduces pings and directory listing udp packets because the clients get significantly shorter lists to ping. Multiple directory servers for a given genre allows for failover and multiple variable pinging to reduce workload from O(n^2) to O(n).

[gene96817]

Also possible, since the server list is relatively stable, a client can/should ping some servers much less than every few minutes. The ping rate can be considered excessive for a stable list. We have many choices to reduce the ping rate.

[gilgongo]

The methods I'm proposing reduces pings and directory listing udp packets

That is an important thing to note, since it aligns to the generally-expressed wishes of most people here to increase the 150 server limit (much as I think that is in fact counter-productive, but I'll shut up about that).