Drew Gallatin, an employee from Netflix, presented the company’s optimization of the FreeBSD network stack at the EuroBSD 2019 conference in Norway. Netflix, which previously pushed for video streaming to reach 200Gb/s on Intel Xeno and AMD EPYC servers, has finally managed to reach 190Gb/s and found that AMD EPYC Naples/Rome servers have the potential to double the potential for up to twice as much as Intel.
Netflix has long known FreeBSD as a data center server system, especially where network performance is demanding. But because they wanted to achieve 200Gb/s network performance on a single server, they started optimizing the NUMA unit on the FreeBSD’s network stack. Allocating space to the kernel TLS key cache on NUMA local memory, as well as using sendfile transfer files, are also within their optimization range. At the same time, Nginx’s network connection processing and inbound connection processing were improved.
Just to see the results here, Netflix’s NUMA optimization on FreeBSD has increased the transfer rate from 105Gb/s to 191Gb/s on Intel Xeno servers, while the NUMA bus occupancy rate has dropped from 40% to 13%.
AMD EPYC’s performance was even more impressive, from 68Gb/s to 194Gb/s. In other words, EPYC was initially much slower than Xeon, but now Netflix’s AMD EPYC server is closer to its 200Gb/s target than Intel’s.
Not only because EPYC is faster, but also thanks to the 128 PCIe transmission channels per slot, they can gain the ability to use two Intel Xeon CPUs in one slot. Netflix, on the other hand, has criticized AMD’s tools for being too scarce (even on Linux).
Eventually, Netflix can now reach 200Gb/s of encrypted video streaming on each FreeBSD server. Details can be found in this slide show.