Jamulus server not visible when using different ISPs

[georg-sch]

Situation: I have set up a Jamulus server (version 3.6.2) on a virtual server hosting (Exoscale, Vienna), using Ubuntu 20.14 LTS. This server is perfectly visible from several Jamulus clients, and they can connect without issues.

Problem is: When I try to switch my ISP to a different connection (e.g. Magenta Webcube), this server does not show up in the servers list of my Jamulus client any more. At the same time, other servers are still visible, and I can connect to them without issues (not going into discussing latency implications of of cable-ISP vs. mobile ISP).

I tried this several times, with varying configurations (different client PCs, different ISPs, cable, ADSL, mobile...), also opened a ticket with the hosting provider, who also tried several things. All result in the same situation: The server wouldn't show up in Jamulus, on certain ISPs, and would show up and can be connected to without issue, with other ISPs.
By the way, the client machine can ping the server machine without issues, and with reasonable ping times all the time.

When I start the client with "--showallservers" parameter, I can see the Jamulus server in the list, but without a ping value.

My assumption is, if the Jamulus client cannot calculate a ping time to the server, the server will be removed from the list of available servers. Is this correct?

Question: How can I access my server from my Jamulus client, when running on (e.g.) a Magenta ISP?
What could be the cause for the server not being accessible?

[ghost]

Try:

• don't allow any firewall to block fragmented UDP packets. (blindly blocking fragmented IP packets is naive)
• use Jamulus client and server >= 3.6.2

[AmyYummy]

尝试：

• 不允许任何防火墙阻止碎片 UDP 数据包。（盲目阻止零碎的 IP 数据包是天真的）
• 使用 Jamulus 客户端和服务器 >= 3.6.2

This is a good suggestion! Much doubt about the fiber .........

[rdica]

404. Be careful with this link.

[hoffie]

Looks like http(s):// was missing and https://www.baudcom.cn was meant instead (I don't know them, but they don't look suspicious at first sight).

[gene96817]

Since ping works, can you get a connection using the server IP address?
It would be interesting to also try to connect the server using the domain name.

[georg-sch]

Thanks for the suggestions.

I tried to connect directly to the server IP address as well - stuck at "Trying to connect", so - not working.

Regarding blocking of UDP packets: If it were the firewall, wouldn't it affect every connection to any server? As it is, with this special ISP, other Jamulus servers are visible and I can connect. Only my server (and I suppose potentially others that I didn't recognize) is neither visible, nor can I connect. And I have no firewall that would specifically block e.g. fragmented UDP packages.
Maybe the ISPs block them...?

Client and server version is 3.6.2.

If I understand correctly, the server ping is calculated not with a regular ICMP package, but with a UDP package, that the server sends to the client, when asked by the Central Server to do so?

I found this detailed explanation of how the connection works: #428 (comment)

Maybe it is still the case here, with randomized UDP ports? Just wild guessing...

[gene96817]

I am wondering if we are trying too hard with this problem.
The key finding... other users can reach the server.
There is a failure mode that might apply>
Consider you have a pool of users, some that can connect and a few cannot.
It would be interesting to check the IP addresses and the corresponding ISPs for each IP address. (More specifically, the ASN numbers).
Scenario 1. If the server IP address or domain name was recently changed (< 48 hours ago); the DNS update could be slow for users on the failing ISP.
Scenario 2: It is possible there is a routing failure (i.e. bad routing table entry) for the user's ISP (ASN). This is more likely the user is on a small ISP.

The only other condition I have observed with a client stuck on "Trying to connect" was with my server maxed out. During this time, Chat also was unresponsive, audio was highly distorted and every user was reporting Overall Delay >500ms. You didn't report any of these obvious signs of server stress; thus Scenario 1 or 2 is possible.

[georg-sch]

Some more test results - and a final statement (for now):
I have tested with another Jamulus server instance of a friend of mine, who is hosting his own private Jamulus server with the same vServer provider. Strangely enough, I could access the Jamulus server without issues with all tested ISPs - cable and 4G. (My friend btw. could access my problematic server with cable connection just fine, but also failed with 4G connection)

So I created another new vServer instance (alongside my existing one), and installed a new Jamulus server there - this time not a publicly registered server, but a private server (getting desperate, I know ... :) ). And the connection to Jamulus worked, from any ISP.

So I instantiated another vServer, again installing a Jamulus server, this time again as publicly registered instance.
Result: I can connect. To both Jamulus servers. Using either cable or 4G internet.

I have now deleted the problematic vServer instance. I will continue to monitor the situation, but for now the systems are running without problem.

All servers: Ubuntu 20.04 LTS 64bit, Jamulus version 3.6.2. built with Simons Jamulus Setup script from
https://worldjam.vip/jamulus/setup-jamulus.sh

[shaggetz]

Today I've solved a related issue, and share my findings here, hoping that it helps you and in other related cases (e.g., #1170, #1478).

For a choir, I run a public Jamulus server on my PC at home, where I have high-bandwidth fiber connection and some routers in front of the PC.
When I only published the server under a given genre in the Jamulus server list, but without port forwarding through all routers and my PC firewall, most clients could see my server as expected. However, a few Jamulus clients would just not see the server in the list. This affected both new 3.8.1 clients, as well as older 3.6.2 clients. It was always the same clients who were affected, so no random factor involved.
I worked around this issue by forwarding the Jamulus port through firewall and routers. Then the clients of all choir members could find my server.
I believe that port forwarding should not have been necessary. I'm not running a private server, as it is listed on the Jamulus central server.
I suspect that it depends on the client's internet service provider, whether a client can see my listed server without port forwarding.
I don't know enough about networks to figure out how some ISPs can prevent the successful ping/connection from client to listed server, which is apparently needed in order for the server to appear in the client's list.
But I hope that the clear linking of this problem to port forwarding may help others in diagnosing it further.

[dcorson-ticino-com]

Public server -> no port forwarding needed.
Private server -> you need port forwarding.

If Public servers are not seen that is an indication of communications problems.
Note that Jamulus uses UDP and not TCP.
Tools testing for TCP communication (for example ping) may not be a good indication of working communication.

[gene96817]

It is difficult to have a specific solution without more details. However some basics ( I'm assuming your server is on your local network behind a firewall).

• For your outbound traffic (server to the world, use port forwarding to ensure your server port numbers are preserved at the firewall. There are some Jamiulus behaviors that could be confusing if the server port number does not match the port numbers on the router.
• For your inbound traffic (users on the Internet to your server on your private network) use port mapping so that the port numbers to your public IP address is preserved at your server's private IP address.

You can make things work without port mapping but it is not worth learning about all the side effects.
If you are trying to have multiple servers on your private network behind an IPv4 firewall, it is possible but it is really asking for a lot of complexity managing all the port mapping. Please don't try this.

[ghost]

More discussion on this topic with an explanation by softins is here #1952 (reply in thread).
This got me thinking of two more possible problems:

• many servers suddenly pinging a single client could look like a DDOS (distributed denial of service attack).
• if the client pings these servers before the servers firewall UDP ports And router UDP ports are Forwarded by the ping they sent to the client... the server will miss the first ping and have to wait awhile for the second.

A solution would be to:

• have the servers immediately ping the client with a TTL of 2 (or so) so that the packet gets past the Forwarding edge router and into the internet gateway, but no further. (no more DDOS)
• have the client wait a second for the fast servers to be ready to receive pings

[gene96817]

@DavidSavinkoff Maybe the Jamulus startup handshake needs a review. Now that I see Jamulus all the way out to Australia, Taiwan, and Asia, the handshake might need to be more tolerant of a lost or very late packet. I am at UTC-10 and I have reported some issues with long latency. A year later, there are more users farther away from Europe. I am not implying the problem in this thread is from a server on the other side of the world. I am observing that that problem could be just the effects of lost or late packets.

[shaggetz]

In the attached file PublicServer_PacketInfo.txt you'll see the server's UDP packets when a client from a problematic ISP connects.

It covers the 2 cases:

• Client fails to connect when Jamulus port is closed on the public server.
• Client connects fine when the Jamulus port is open on the public server.

Even when the client fails, the opening of the server port with the first STUN-like packet seems have worked, because the server receives multiple packets from the client.
I hope the timestamps can give you an idea whether the issue is related to late packets, or if a "bad ISP" means something else.