Central servers: issues and possible approaches

[gustebeast]

The server documentation says that "Default" has been deprecated in favor of "All Genres". Can we merge these two lists in the client?

[pljones]

The reason they were split still exists.

At the moment, the list can get fragmented if too large and, where UDP fragmentation causes routing issues, this can cause the list to become only partially visible.

I'd also note that All Genres is pretty close to full, so something is going to need to be done one way or another.

As a further point, having the client poll a very large list of servers, most of which they won't be using is a waste of resources both on that client and on every server in the list.

[gustebeast]

It feels strange that we would use genre as a means of solving this scalability issue. Normally I would rely on pagination for that. The client only gets 50 servers or so from the server, then if they want they can request the next 50 and so forth.

The problem is that we can't send them the 50 servers with the lowest ping since the client needs to compute the ping for all the servers before it can sort them. Would it be possible to sort by distance instead of ping? The user experience should be comparable even if the ordering is slightly different. We can get/cache the locations using the client and server IPs. This would also get rid of the ordering issues when pings fluctuate, and the time required for the servers to sort when switching genres.

I'm also curious how much of this scalability issue is client vs server (jamulus.fischvolk.de).

[pljones]

Would it be possible to sort by distance instead of ping?

Distance from the server list? That wouldn't help much. To sort per client, it would have to have some way to compute -- on every client request, every two seconds -- the distance between the client and every server. That sounds like a lot of work. And physical distance isn't much use when it's the ping time that's important. I get half the ping to Frankfurt than Cardiff from London. Even Zurich is better than some places in England.

[gustebeast]

Sort per client yes. We wouldn't necessarily need it to update every two seconds though. The initial request (to central server) would just run once (with a refresh button to re-query). It would require looping through every server, but if you can cache the IP -> location mapping, then you'll just have to do a bit of basic math per server per request: https://stackoverflow.com/a/27943. You may even be able to avoid checking every server if they are stored in a data structure that is organized by location.

Once the client has the 50 servers, it can poll them every 2 seconds for the ping/musicians.

For the Frankfurt vs Cardiff example, my hope is that all those servers would appear near the top of the list, so while the lowest ping may not appear first, it would be easy enough to find.

[gilgongo]

The genre-based system is not ideal and has some clear disadvantages. So if the CS limit issue could be solved in a different way that would be nice.

so while the lowest ping may not appear first, it would be easy enough to find.

That's an an interesting observation I think because most people will only have a handful of fast enough servers to use anyway, and once they find one they're probably not going to care about sort order after that. So a "rough" sort on location (however that's done) should be fine even in a list of 50 I would think. Similarly with a fresh button: servers and players don't churn very fast, so I doubt you'd need to hit the button more than a couple of times before joining a server for a jam.

I take it also that servers would attempt to register on all central servers and go with the first free slot they were given?

[gustebeast]

with a fresh button: servers and players don't churn very fast

It may be advantageous to separate the query that gets the 50 servers from the query that checks players. With the current system, does each server report the players to the central server, then the central server reports the players to the client, or does it work like ping where each client asks the server directly?

I take it also that servers would attempt to register on all central servers and go with the first free slot they were given?

What do you mean by a "free slot"? The simplest set up would be to have one central server with a single unified list of servers. Depending on the load it gets, this could be done with a single beefy computer or require load balancing between multiple computers (still with a single domain and shared unified list). Load balancing may be tricky, although we could tackle it piece by piece, start by implementing the distance sorting, then unify the genres.

We need a way to avoid the inevitable issue of a genre having too many servers in it (unless we want to have a jazz1 and jazz2). However, I think this proposal does more than that: it paves the way for some nice improvements to the user experience:

1. User opens Jamulus for the first time, opens server list.
2. User is quickly presented with the 50 closest servers. The server name, location and genre columns appear right away, the ping and musicians populate after a second or two (and continue to update every couple seconds).
3. The user opens up the "Genre" filter dropdown and selects "Jazz" and "All Genres". The list then updates to show the 50 closest servers that have their genre set to "Jazz" or "All Genres".

If the user opens Jamulus back up, it would remember the filters so the servers would appear in the same order right away. We could still support filtering by server name or occupied, the server would just find the 50 closest servers that match the name/have musicians.

[gene96817]

Please don't implement a "nearest 50 server" list. I recently moved across 5 timezones. I now have Jamulus friends across 8 timezones. I won't see my friends from my old place with only the nearest 50 servers displayed. Also, my old friends won't see my new server and visit. It has been really great to have old friends dropping in, even if the total delay is >140ms.

[gene96817]

I just checked Jamulus Explorer and there are 78 servers under Genre Classical/Folk/Choir. My client shows 38. Any clue why the number of servers are different?

[ghost]

Maybe Jamulus should display all servers (ip addrs Or fqdn Or whatever can be displayed) in the list before pinging them instead of not displaying the unpinged servers. Then the list can be sorted by the client as it pings (the way it is now). I think adding heuristics etc. is just adding bloat.
Edit:
I just found another Undocumented Jamulus feature that solves this problem:
Jamulus --showallservers
please add this documentation to Jamulus --help AND the documentation page.

[gustebeast]

Would these options be sufficient for your use case?

1. At the bottom of the connection window, there would be a "Next Page" button so you can see servers that are further than the closest 50.
2. We could add the ability to "favorite" a server so you could quickly see those even if they span large distances.
3. You could filter out empty servers to keep the list under 50 (provided you are looking for online friends).

Additionally, we can make the page size larger than 50. Could probably make it the current max server count per genre without hurting performance.

[gustebeast]

The problem with sorting by ping is that you need to ping every server in the list. Creating separate lists per genre has worked for now, but that isn't a long term solution. We shouldn't need to separate "default" and "All Genres", and it should be possible to see all genres in one list if you don't care about what genre you want to play/listen to. This is made especially painful because it takes a while for the servers to sort when switching genres.

The location heuristic I think has the potential to actually remove bloat both in the user experience and in the code. For example, we could get rid of the logic that prevents ping sort order changes on mouse hover.

[gene96817]

@gustebeast I would prefer the connection window scroll. "Next" would work also work. It would be nice for the connection window to indicate the number of servers detected. Keep the list simple and just sort by ping times.

[gene96817]

If you don't sort the server list by ping, how will I know how long it will take my packets to reach the server? Also, the ping times could change by network traffic (e.g. time of day). Of course, we don't want the list to churn because of jitter in the ping times. I am sensitive to this issue because I don't live in the center of a metropolitan city. Locations at the edge of the city or further see more delay from fluctuations in network trafic.

[gustebeast]

The ping would still be displayed in the same place. It's just a matter of how the list is sorted. If distance is a good enough approximation of ping, you'll still see the low ping servers at the top of the list. Per pljones' example above, there will be cases where a server with a lower ping appears lower in the list, but so long as it is still close to the top, it should be easy enough to look at the pings and pick out the best one.

[gene96817]

What if the client keeps a single list of all servers, only the selected subset would be displayed. I agree that we don't want to be pinging all the servers frequently. We could ping the serves in a slow background process. If we keep the list between sessions, the old data should be a good starting approximation. Then only new users starting the client for the first time would need to have all the servers pinged.

I often see some servers that are farther away with shorter ping times. It depends on whether those servers are close to a major backbone.

[corrados]

The server documentation says that "Default" has been deprecated

I don't know who wrote this but this was not my intention. The two lists are hosted on different server hardware. If one server fails/crashes, the other list is still available. So it makes sense to have them both.

[gilgongo]

I think it was "deprecated" in the sense that "All/Any" genre was the new default. Which would make sense. But it was too long ago to remember the discussion.

[gilgongo]

@gene96817

My client shows 38. Any clue why the number of servers are different?

My clients usually only show about a quarter of the total list. Not sure why but I assume it's an ISP/network thing.

please add this documentation to Jamulus --help AND the documentation page

I think we've kept --showallservers undocumented because we don't want to imply it's some kind of usage option. And it's a bit hard to explain what it's for unless you're an engineer and using it to debug something, in which case it's there in the code. If someone can usefully explain in layman's terms what it does and why it would be useful then perhaps we could document it?

[gene96817]

@gilgongo I haven't yet made any time to look at the code, If you can point me at the code involved, I could look to see if I can find clues to this discrepancy.

Assuming the pings are using UDP, my first guess is that some UDP packets are getting lost or getting back very late. Both behaviors are "normal" for UDP and the code needs to have logic to handle the unreliable nature of UDP.

[gilgongo]

@gene96817 I'm not sure, other than it being mentioned here

[gustebeast]

I don't know who wrote this but this was not my intention. The two lists are hosted on different server hardware. If one server fails/crashes, the other list is still available.

It is written here: https://jamulus.io/wiki/Central-Servers
Load balancing should be an implementation detail that is invisible to users, the user interface should be designed based on what is intuitive and convenient, not based on technical limitations. It is possible to have multiple servers serving the same unified list, so if one goes down it just increases the load on the other servers but the user doesn't necessarily notice. Server operators shouldn't need to change their genre in their config to deal with outages.

If we keep the list between sessions, the old data should be a good starting approximation.

This could work, and has the advantage of not needing more compute power on the central server. However, I think it could lead to some confusing experiences. If you launch Jamulus for the first time, but forget to turn off your VPN, the initial ping estimates may be incorrect. If you then notice that your VPN was on, turn it off and relaunch Jamulus, it may take a while for a server right next to you to get pinged and appear at the top of the list.

I often see some servers that are farther away with shorter ping times.

pljones also noticed this. My thought is that it is okay if a lower ping time appears lower in the list so long as it still appears close to the top, since you can look at the pings and pick out the best one. So if servers 1 and 2 are closer but higher ping than server 3, not a big deal. However, if servers 1-10 are closer but higher ping than server 11, then this assumption breaks down somewhat.

[ghost]

Jamulus --showallservers
Run this command to "actually" show all servers with the capability to sort list on Server Name, Ping Time, Musician or Location.
This command is for all engineers, including train engineers and non engineers.
TRY IT.
BTW an improvement for this command would be to have the unpinged and unresponsive servers at the end of the list.

Jamulus --showallservers
is so good that it should be the default.

The current list in "Connection Setup" should be an undocumented:
Jamulus --showrandomservers

[corrados]

Jamulus --showallservers is so good that it should be the default.

If you cannot ping a server, you cannot connect. So it does not make sense to show these server since that only confuses the Jamulus users. --showallservers is just for debugging.

[gene96817]

@gene96817 I'm not sure, other than it being mentioned here

Thanks for the reply. I only saw the code decide if debug mode is on to display all servers. The client code that would be interesting would cover (1) the ping test, and (2) how to decide which server not to display. This you already know. It is the obvious question for the code. I am most curious about the ping test.

[gustebeast]

Getting back to the subject of this issue, I still think the current genre filtering and ping sorting pose a significant problem both in the user experience and our ability to scale Jamulus.

If you only need to look at one genre, the UI works pretty well, but switching genres is a major pain. The ping takes several seconds to sort, and if you hover your mouse over the list too soon, it will stay unsorted, which isn't particularly intuitive. To ensure the client isn't pinging more than 150 servers, we have a cap on the server count per genre, which forces the user to choose between "default" and "all genres" without any context as to what each is for. This cap is just a temporary fix, and unless we want to make a jazz1 and jazz2, we need something to address this issue.

Sorting by ping does a great job at putting the servers you likely want to join at the top of the list, but it won't scale unless we want clients to be pinging a large number of servers every few seconds. The proposal here is to use location sorting, which will hopefully do a good enough job at putting the right servers at the top of the list, even if the ordering doesn't fully match up with ping times. For that sacrifice, we can build a more functional UI (multi genre selection) that loads quickly, sorts consistently, and obfuscates implementation details like default vs all genres.

Here is an updated step by step implementation that avoids changes to the central server code:

1. Update location sorting when using --showallservers to sort by distance rather than city name.
2. Change default sorting from ping to location.
3. Add pagination. If the list has more than 150 servers (current cap per genre), a next page button appears at the bottom to see the next 150 servers,
4. Replace genre selector with a multi select input where default and all genres are merged into a single checkbox called "all genres". The client will send a request to each genre's central server, collect the responses into a single list, sort by location, and display the first 150.

This solves the scalability issue of pinging too many clients and makes browsing multiple genres easy. If I like all genres, rock, and jazz, I can see them in one place right away instead of having to go through the switch genre -> wait for ping -> switch genre loop.

[corrados]

This is made especially painful because it takes a while for the servers to sort when switching genres.

I have improved the server list recently (4310fe5). Here is a comparison:

Current official release version Jamulus 3.6.1:

Git master code after recent change:

As you can see, the list updates the ping times much faster so you very quickly get the servers near you displayed now.

[gene96817]

This developer community is awesome. I can barely keep up with just monitoring the discussion.
Here is how I am understanding the discussion on the server list. I hope there are a few ideas here that will be useful.

This scheme:
1- does not expect any changes to the central server
2- reduce the number of pings needed
3- hopefully simplify the management of the server list.
4- we expect servers do not move physical locations
— yes it is possible that a server is frequently moved and this will be a stress (but not break) this scheme

Consider:
— the client maintains a list of previously seen servers.
— server-name
— IP address
— genre
— last ping time
— operations-status up/down
** optional/extra data (see discussion below)
— last seen
— min and max ping time
— number of times visited (aka favorites)

— client start-up
— the client gets a list of operating servers from the central server and merges it with the list of previously seen servers.
— never seen servers are added to the list
— the up/down state is updated
— Notes:
— the genre selection determines which server entries are displayed
— at this point, we have a usable approximation of what servers are available.
— if the global network was perfect, the old ping times are a good starting point for the servers that are up (actually for a perfect network, the ping times never change)
— start-up phase ping update
— we have many choices of heuristics of which servers should get pings updated
— example: get updated pings for the servers with the biggest spread between min and max pings
Because they probably are at places that have bigger congestion problems
— example: get updates of highly visited servers (because the user is more likely to connect to those servers)
We want to flag highly visited servers as down so the user does not try to connect to a down server and get frustrated
— Limit the pings to minimize global ping traffic
We want to manage the number of start-up pings and then have a low rate of maintenance pings
— there is only one server list, we only display the entries selected by the genre filter,
there is no churn caused by changes to the genre filter.
— we can choose to slow some client start-up processes so that the start-up ping process is done before the user has a chance to look for a connection

— client post-start-up
— the client sends a low rate of pings to update the table
-- a slow rate pinging all the servers (round-robin) works
— or, there are many heuristics we can choose to prioritize update rates
— example: ping the server with the oldest last seen date first
— example; ping servers with the greatest variance of min/max pings
— now we can reduce global pings to any level we want.
— is one ping per minute too slow… that would update the whole table in 4-10 minutes
— with the last-seen, we can create a policy of when to decide a server should be removed as retired
— it is easy to add a server back. However, removing the server loses the favorites count.
-- it is necessary to remove servers so that the list doesn't accumulate permanently retired servers.
— we can now detect servers with poor internet service (high packet loss or high congestion)
— or we can choose not to care,
— being able to frequently update ping times on a server with congestion will show the user fluctuating ping times.
— we get to decide how much ping traffic we are willing to give to this concept.
— if we don’t mind some complexity, if a server does not respond to a ping, and the min/max spread is large,
Use multiple pings to determine the server status (because that server is more vulnerable to have packet loss).

[gustebeast]

I have improved the server list recently

Looks great! That should fix most of the gripes I have with the current browser as a multi genre user.

We still have the problem of default vs all genres though, and the 150 server cap per genre, which will only get more problematic as Jamulus grows. We need some way to list more than 150 servers in the client.

Gene's approach above could work. We ping all the servers the first time, but then cache the results and find some heuristics to avoid repinging every server every time. Maybe we can have a check for the client's IP at start up so if you are on VPN and then turn it off, it will know to clear the ping cache. This won't scale quite as well as location sorting, since we'll still need a lot of pings, but could likely last us a while, and potentially be sufficient for the lifetime of Jamulus.

[gene96817]

I don't know if there is a good solution for VPNs, especially for people that also use their computers for their jobs. We should make it clear that it is better for the VPNs to be turned off. There are some people who use a split-horizon with their VPNs. In this case, they should make sure their Jamulus traffic is not sent through the VPN. The good news here is those users are pretty knowledgeable.

I hope we don't decide to use distance to sort servers. It might work in Europe. In North America, I have seen ping times vary significantly (15-20ms) when the apparent server separation is less than 200km. It is even more pronounced in my current location UTC-10. This is probably because the route of the major fiber runs matter.

I am confident we can design a ping algorithm that can keep the flow of pings low for 1000 servers. If most of the servers have stable ping times, 5 or 10 minutes for a refresh should not be awful. For the most part, we want to detect when a server turns up or shuts down. Even if there were a mean of 50 users per server, the number of users starting up in the same 5-minute interval should be pretty small.

[chrisrimple]

Sorry for the late comments, but I wonder if we're asking the wrong question? The use cases for needing a server list include:

1. Musician A wants to play with Musician B, they already know (or know of) each other, and are trying to find a low latency server on which to play
2. Musician C wants to play with anyone, and is trying to find a (optimally low latency) server on which there are already other musicians
3. Listener D wants to hear others playing, and is trying to find a server (latency doesn't matter) on which there are already musicians playing

To satisfy each of the use cases, I need:

1. A list of low latency servers. Since distance is generally fixed, latency doesn't change much between sessions (but can). So an out-of-date-but reasonably-accurate list of servers and ping times is fine. Yes, it might be showing a server that's not up, or missing a new server, but is still 90%+ accurate. It could be made more accurate with a background update after rendering the initial list from a (local) cache.
2. A list of servers with musicians on them. Like (1), a (local) cached list is going to be 90%+ accurate for servers and ping times, but won't be accurate for musicians on those servers. So maybe queries for servers + ping times should be handled separate for queries for musician occupancy lists?
3. A list of servers with "active" musicians on them. Essentially identical to (2), unless we can improve it in other ways.

What else can we do to make the server list faster, easier to use, and more valuable? How about:

4. Add a checkbox for "Only servers that are in use", which is the same as putting # in the Search field. Enable the checkbox on initial install, then store the state in the INI file. Use cases (2) and (3) will benefit greatly, and maybe a query for servers + ping times only needs to return results for servers where number of users > 0.
5. Add a checkbox for "I'm jamming, join me" to the Settings, Profile, or somewhere, so that client can be counted as "active" for use case (3), then show a count of "active" musicians in the server list, or otherwise highlight the server as "open for joining right now" when "active" musicians > 0.
6. Allow a user to mark a server (or servers) as Favorites, so they always appear at the top of the server list and are not sorted by ping time. This solves use case (1) when the musicians are regularly meeting on the same server (including a server that they may own).

Sorry that I can't help code these solutions. :(

[gene96817]

@chrisrimple It seems the prior discussion covers #1 - #4.
For #5, the moment you are on a server for "I'm jamming, join me" this becomes the same as #3.
For #6, in normal conditions for your use case, I think the favorite servers will be low latency and near the top of the list. The unusual case would be when I regularly jam with my old friends 5000+ km away. In that case, because of friendship, we ignore the terrible latency. This terrible latency puts the server deep in the list. However, it shouldn't matter because this favorite server is remembered in the Server Address dropdown box.

[corrados]

is one ping per minute too slow… that would update the whole table in 4-10 minutes

The server list was never meant to be opened that long. You click on the Connect button, select a server and jam. BTW, you should not have the server list open during a jam since that may cause audio drop outs.

If you want to monitor the Jamulus servers for a longer period of time, the Jamulus Explorer is a better choice.

[gene96817]

Good to know the server list could cause audio dropouts.
It is even better if the server list is mostly closed. Maintaining (updating) the table in the background is a good way to keep the ping traffic to a minimum. Then displaying the list and changing the genre selection doesn't create additional ping traffic.

I find Jamulus Explorer interesting because some servers on its list don't show in my client. I don't yet understand this discrepancy. Other than that Jamulus Explorer shows the Jamulus universe from 10 timezones away. That view of latency is not my world. I am on at edge of the universe.

[drummer1154]

If you want to monitor the Jamulus servers for a longer period of time, the Jamulus Explorer (https://explorer.jamulus.io/) is a better choice.

OK, but the ping RTTs in Jamulus Explorer are not related to my location (Munich, DE), but to London, UK. RTT significantly depends on the transport media - fiber is a lot more performant than copper. Therefore "distance" should not be a sort criteria.

[gilgongo]

Has this ticket reached a conclusion or resolution we can action? If people object to describing "Default" as deprecated in favour of "All Genres" can somebody suggest another way of describing it here? https://jamulus.io/wiki/Central-Servers The current server GUI defaults to "All Genres" so I'm not sure how to describe it.

[passing]

@gilgongo thanks for pointing me here!

I am trying to combine the ideas i found in this issue together with my own view (#875 (comment)) but am also reducing this to the - in my opinion - most relevant points.

Current architecture disadvantages

The current architecture with multiple central servers has several disadvantages:

• With more central servers, users need more time to find the servers they are looking for, while pinging many servers not relevant for them
• When central servers are added, a new Client version needs to be released that all users need to update to.
• When central servers are added, server Administrators need to change the central server in their configuration when genres got split
• In case your server has some downtime, you can lose your slot and have to reconfigure your server to use a different central server.

By changing to a single central endpoint (a single central server, or multiple servers that are in sync) you would get rid of these issues and have many more possibilities for improving the user experience in regards to server filtering (see below)

Current architecture advantages

• The different central servers are hosted on different server hardware. If one server fails/crashes, the other lists are still available.

Here's actually the question about the risk when using just one central server: I guess losing data on the Central Server is not an issue as it will be recovered automatically by re-registrations of all running Servers. Also a short downtime of the Central Server would not affect any ongoing session.

Anyway, you may want to have a fallback that kicks in immediately. But that would require that the data can be synced in some way.

Requirements from user perspective

Location filtering

Users want to find servers that are close to them. The ideal indicator for that is the ping, but that can just be determined by the client itself. It would be way easier to be able to filter by Location (Continent or single/multiple Countries), so the client just sees Servers based on the Country in their Server information.

Genre filtering

If the genre would be another field in the Server information (e.g. a genre ID), that could be used for filtering without having to use multiple central servers for separation of genres. That would also allow to add other genres easier later.

Server name filtering

already possible

Utilization filtering

Right now you can filter for servers that have users connected, that could be extended to servers that are empty.
A separate control (e.g. checkboxes) would be nice instead of reusing the search field (#)

Favourites / Last Used

Would be nice to have.

Considerations on filtering

Where to filter

Filtering could be done by the server, or the server could send the complete list to the client and the client would apply the filters and just display the results. Not sure which variant would be better.

Reducing pings

Caching of ping times was suggested and could be useful, but may be quite complex and has some disadvantages as old pings can be inaccurate.

An easier alternative for reducing pings could be to only ping the servers in the list for a short time (e.g. 1 minute) and have a refresh button (also some keyboard shortcut) to start pinging all servers again. That button should indicate (be active) when pings are ongoing.

Pagination

Of course, there should be a limit on how many servers are listed. That may require pagination if filtering does not allow for getting below that limit.
At the moment the biggest groups are probably:

• Any Genre 1-3, Germany < 50
• Any Genre 1-3, United States ~ 80

So implementating pagination could be done later if it is considered acceptable that you just need to add enough filters for now.

[gene96817]

The only issue for me is that my community is far from all the central servers so the pings seen by the central servers are not representative for our location. While cached pings could be inaccurate (e.g. a server is down), a slow ping update will eventually provide an accurate update.

[gilgongo]

@passing Good summary!

One dimension I'd add is the long-standing issue of too many empty servers.

The vast majority (like 90%) of public servers are empty most of the time. But it's not obvious why this is. I would guess it's because of a combination of (at least) two things:

1. A mis-match between client and server locations. So if you're in Mumbai but nobody's running a server there.
2. The "ghost town effect". So I'm in London where there are 10 servers, but they're all empty so I don't want to join any of them. A vicious cycle which then reverses once two or more join a server. But then the numbers rocket to chaotic levels that make people leave.
3. The ease with which you can set up a public server (start server, give it a name and location and it's instantly listed)

If the above hypotheses are correct (and there may be others), then any given central server solution might want to address them by, for example, discouraging the setup of yet another server in London, and encouraging people to set one up in Mumbai. You could tackle the first with a technical solution (eg limit the number of servers per location and/or impose some entry requirements) and the second with a social one (appeal for servers in poorly-served locations, or semi-automate that in some way).

BTW one minor point:

If one server fails/crashes, the other lists are still available.

The central servers are currently split across only two physical machines (on different networks).

@gene96817

my community is far from all the central servers so the pings seen by the central servers are not representative for our location

The ping times you see in the list are generated by your client, not the central server (you'll notice that the list can take some time to load because of this). So those ping times are accurate for your location. If you are not seeing low-latency servers listed, then see my point above.

[gene96817]

Thanks.... I am concerned that some discussion about ping times relies on the central server's view.

[gilgongo]

Indeed. Any central server solution that involved the central servers themselves handing out ping times to clients would be automatically wrong as far as I understand.

[gilgongo]

Hi all - this seems to be more of a general discussion about central servers. Until we have some agreed definable action to take, I'll move it to a discussion (and update the title) if that's OK.