This project is designed to compare websocket servers in multiple languages and frameworks and has a companion blog post. The servers all implement an extremely simple protocol with only two messages: echo and broadcast. An echo is returned to the sending client. A broadcast is sent to all connected clients. Both messages take a payload value that should be delivered to the appropriate destination.
Example broadcast message:
{"type":"broadcast","payload":{"foo": "bar"}}
For the platforms with low level websocket implementations the above message would work directly. For platforms with higher level abstractions such as Phoenix and Rails the message must be encoded to be compatible with their message standards.
The following platforms currently have servers implemented.
- Clojure
- C++
- Elixir / Phoenix
- Go
- Haskell
- Java
- Javascript / NodeJS
- Ruby / EventMachine
- Ruby / Rails
- Rust
Some dependencies are tracked via git submodules. First step is to pull them down.
git submodule init
git submodule update
Look for a README.md in each projects directory for instructions on building and running the servers.
As part of this comparison a benchmark tool websocket-bench was built to test the performance of these websocket servers. websocket-bench is designed to find how many connections a server can handle while providing an acceptable level of performance. For example, given the requirement that 4 broadcast requests are served concurrently and 95% of broadcasts be completed within 500ms, how many connections can the server handle?
Here is an example benchmark run:
% % bin/websocket-bench broadcast ws://earth.local:3334/ws --concurrent 10 --sample-size 100 --step-size 1000 --limit-percentile 95 --limit-rtt 250ms
clients: 1000 95per-rtt: 47ms min-rtt: 9ms median-rtt: 20ms max-rtt: 66ms
clients: 2000 95per-rtt: 87ms min-rtt: 9ms median-rtt: 43ms max-rtt: 105ms
clients: 3000 95per-rtt: 121ms min-rtt: 21ms median-rtt: 58ms max-rtt: 201ms
clients: 4000 95per-rtt: 163ms min-rtt: 30ms median-rtt: 76ms max-rtt: 325ms
clients: 5000 95per-rtt: 184ms min-rtt: 37ms median-rtt: 95ms max-rtt: 298ms
The above benchmark starts by connecting 1000 websocket clients to ws://earth.local:3334/ws. Then it sends 100 broadcast requests with a concurrency of 10. It increases by 1000 clients at a time until the 95th percentile round-trip time exceeds 250ms.
Run make to build the benchmark tool. Ensure you have previously initialized git submodules or the build will fail. The benchmark tool will be built to bin/websocket-bench.
Most servers have sufficient performance to encounter OS level open file limits. Here is how to increase those limits.
Add the following to /etc/sysctl.conf:
fs.file-max = 2097152
Add the following to /etc/security/limits.conf:
* soft nofile 1048576
* hard nofile 1048576
In the shell run (or put in .profile or the like):
ulimit -n 1048576
It is highly recommended that websocket-bench and the server be run on separate machines connected with at least GB ethernet.
Run websocket-bench with the --help parameter for detailed info.
% bin/websocket-bench --help
A host can only establish a few ten-thousands of outbound connections before it suffer port exhaustion. To be more accurate that limit is per IP address. websocket-bench can use multiple IP addresses to establish more connections.
bin/websocket-bench broadcast ws://earth.local:3334/ws -c 4 -s 40 -l 192.168.50.5 -l 192.168.50.246 -l 192.168.50.247 --step-size 1000
The above command would use addresses 192.168.50.5, 192.168.50.246, and 192.168.50.247.
Of course, this requires that the host have multiple IP addresses. On Ubuntu 16.04 additional addresses can be bound to an interface by adding configuration to /etc/network/interfaces (this may require disabling network-manager if the machine is a desktop installation).
Example /etc/network/interfaces snippet:
...
up /sbin/ip addr add 192.168.50.246/24 dev eth0
up /sbin/ip addr add 192.168.50.247/24 dev eth0
down /sbin/ip addr del 192.168.50.246/24 dev eth0
down /sbin/ip addr del 192.168.50.247/24 dev eth0
...
Results are in the results directory.
This project is complete and is no longer actively maintained. We'll leave pull requests demonstrating other implementations open for educational purposes.
websocket-shootout is supported by the team at Hashrocket, a multidisciplinary design and development consultancy. If you'd like to work with us or join our team, don't hesitate to get in touch.