Made a quick test of mesh VPN clients. Test was performed between host and a VM, both running Kubuntu 23.04. VM ran on KVM with a virtio network adapter.
Test machine is oldish laptop with an i5-2540M, so VPN performance was probably CPU bound. Still, tests help to understand how different mesh VPNs compare against each other.
Tailscale surprisingly was the fastest, even faster than plain Wireguard, despite being userspace. But it also consumed more memory (245 MB after the iperf3 test!) and CPU.
Netbird's CPU usage is so low I almost doubt if that's fair comparison, most usage might be by kernel, since it uses kernel Wireguard. I don't know how to measure that better. Memory usage is moderate. For some reason it wasn't as fast as plain Wireguard.
Zerotier has the lowest memory usage, but is the slowest. Although this would probably only matter on LAN. Hope upcoming v2 closes the gap.
| Tailscale | Netbird | Zerotier | Wireguard | Raw | |
|---|---|---|---|---|---|
| Version | 1.48.2 | 0.23.3 | 1.12.2 | ||
| Idle, PSS, MB | 66 | 36 | 12 | ||
| iperf3, PSS, MB | 245 | 36 | 12 | ||
| Idle, CPU time, s / real minute | 0.505 | 0.120 | 0.297 | ||
| iperf3, CPU time, s / real minute | 115.23 | 0.14 | 78.72 | ||
| iperf3, Mb/s | 860 | 630 | 360 | 730 | 9600 |
Same table as an image with best / worst results highlighted
UPD: since this got more attention, questions and suggestions than I expected, I've made more tests.
| Tailscale | Netbird | Zerotier | Wireguard | Raw | |
|---|---|---|---|---|---|
| Pre / post start, ฮ mem avail, MB | 48 | 43 | 12 | 22 | |
| Pre / during iperf3 tests, avg ฮ mem avail, MB | 130 | 8 | 8 | 7 | 9 |
| Total mem usage under load (sum of the above), MB | 178 | 51 | 20 | 29 | 9 |
| CPU usage during iperf3 tests, avg % | 62 | 77 | 57 | 77 | 27 |
| iperf3, Mb/s | 935 | 946 | 414 | 988 | 10340 |
| iperf3 bidirectional, total, Mb/s | 427 | 853 | 480 | 1029 | 10260 |
Same table as an image with best / worst results highlighted
This time I've measured total system memory usage to account for kernel usage. No surprises here, just a bit more data. Tailscale confirmed to consume a lot more memory under load.
How I've measured memory and CPU usage:
- All commands and measurements are done in the VM, the only thing on host is VM itself and iperf3 client
- Boot the VM, start iperf3 server, all mesh VPN services are disabled beforehand
- Then, for every service:
- Run
sar -r ALL 2 5to determine the baseline, note the averagekbavail(it accounts for the fact that not all caches are actually reclaimable, see man) - Start the service
- Run
sar -r ALL 2 5, note the averagekbavail, difference with the baseline goes to the "Pre / post start, ฮ mem avail" row - Run iperf3 client on the host:
iperf3 -c *IP* -t 90oriperf3 -c *IP* -t 90 --bidir - During the test, run
sar -u ALL -r ALL 5 10, note the averagekbavailand CPU idle % - After all tests are done, average of all
kbavailduring tests goes to the "Pre / during iperf3 tests, avg ฮ mem avail" row; and 100% minus average of all CPU idle % during tests goes to the "CPU usage during iperf3 tests" row - Stop the service
- Run
sync; sudo sysctl -q vm.drop_caches=3
- Run
Here's raw data, so you can recalculate using anything else instead of kbavail
I've minimized number of running apps on the host that could affect performance (looking at you, Firefox). This resulted in some overall performance increase, and both Netbird and Wireguard performed almost identically to Tailscale in iperf3, overperforming Zerotier by ~2.4 times (the same ratio as in my initial test).
I've also added bidirectional iperf3 test, so that both ends transmit and receive data. That didn't significantly affect the performance, except for Tailscale. Tailscale performance halved in this test.
After 10 minutes PSS was back to ~74 MB