TL;DR

We'll be optimizing these key areas:

  • NIC offloading techniques
  • TCP_QUICKACK for reduced latency
  • net.core.rmem_max tuning
  • SO_BUSY_POLL for CPU efficiency
  • Strategies to minimize latency spikes

The Need for Speed: Why 100Gbps?

Before we dive into the nitty-gritty, let's address the elephant in the room: Why 100Gbps? In the world of high-frequency trading, every microsecond counts. We're not just talking about bragging rights here; we're talking about the difference between making millions and losing your shirt.

But achieving and maintaining 100Gbps throughput isn't just about throwing more hardware at the problem. It's about fine-tuning your system to squeeze every last drop of performance out of your existing infrastructure. And that's where kernel tuning comes in.

NIC Offloading: Let Your Hardware Do the Heavy Lifting

First things first: If you're not leveraging NIC offloading, you're leaving performance on the table. Modern NICs are capable of handling many network-related tasks that would otherwise bog down your CPU. Here's how to check your current offload settings:

ethtool -k eth0

Look for these key offloads:

  • tcp-segmentation-offload (TSO)
  • generic-receive-offload (GRO)
  • receive-side-scaling (RSS)

To enable these offloads, you can use:

ethtool -K eth0 tso on gro on

But wait, there's more! For 100Gbps networks, consider enabling these advanced offloads:

  • ntuple filtering
  • receive packet steering (RPS)
  • receive flow steering (RFS)

These can significantly reduce CPU usage and improve packet distribution across cores.

TCP_QUICKACK: Because Patience is Not a Virtue in HFT

In high-frequency trading, waiting for ACKs is like waiting for paint to dry – ain't nobody got time for that. Enter TCP_QUICKACK. This nifty little option tells the kernel to send ACKs immediately, rather than delaying them.

To enable TCP_QUICKACK system-wide:

echo 1 > /proc/sys/net/ipv4/tcp_quick_ack

For a specific socket in your application:


int quickack = 1;
setsockopt(socket_fd, IPPROTO_TCP, TCP_QUICKACK, &quickack, sizeof(quickack));

Keep in mind that while this can significantly reduce latency, it may increase network traffic. As with all optimizations, measure before and after to ensure it's benefiting your specific use case.

Tuning net.core.rmem_max: Size Matters

When it comes to receive buffers, bigger is often better – to a point. The net.core.rmem_max parameter sets the maximum receive socket buffer size in bytes. For 100Gbps networks, you'll want to crank this up:

sysctl -w net.core.rmem_max=16777216

This sets the maximum receive buffer to 16MB. But don't stop there! You'll also want to adjust these related parameters:


sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"

Remember, these changes are temporary. To make them permanent, add them to /etc/sysctl.conf.

SO_BUSY_POLL: When Busy-Waiting is a Good Thing

In the world of low-latency networking, sometimes the best way to wait is to not wait at all. That's where SO_BUSY_POLL comes in. This socket option allows the kernel to busy-poll for incoming packets, rather than relying on interrupts.

To enable SO_BUSY_POLL in your application:


int busy_poll = 50; // Time in microseconds
setsockopt(socket_fd, SOL_SOCKET, SO_BUSY_POLL, &busy_poll, sizeof(busy_poll));

You can also enable busy polling system-wide:


echo 50 > /proc/sys/net/core/busy_poll
echo 50 > /proc/sys/net/core/busy_read

Be cautious with this setting, as it can increase CPU usage. It's best used on dedicated networking cores.

Taming the Latency Beast: Strategies for Reducing Spikes

Even with all these optimizations, latency spikes can still rear their ugly heads. Here are some additional strategies to keep them at bay:

1. IRQ Affinity

Ensure that network interrupts are handled by dedicated CPU cores:


echo 2-3 > /proc/irq/YOUR_ETH_IRQ/smp_affinity_list

2. CPU Isolation

Isolate CPUs for your critical networking tasks:


isolcpus=2-3 nohz_full=2-3 rcu_nocbs=2-3

Add these to your kernel boot parameters.

3. NAPI (New API)

Ensure NAPI is enabled on your network interface:


ethtool -k eth0 | grep "napi-tx-"

4. Tune the Scheduler

For latency-sensitive tasks, consider using the SCHED_FIFO scheduler:


struct sched_param param;
param.sched_priority = 99;
pthread_setschedparam(pthread_self(), SCHED_FIFO, ¶m);

Putting It All Together: A Holistic Approach

Remember, optimizing for 100Gbps microservices isn't just about tweaking individual settings. It's about taking a holistic approach to your entire system. Here are some final tips to tie it all together:

  • Profile your application to identify bottlenecks
  • Use tools like perf, flamegraphs, and eBPF for deep insights
  • Consider DPDK or kernel bypass techniques for extreme performance
  • Don't forget about your storage I/O – it can be a hidden bottleneck
  • Regularly benchmark and monitor your system to catch regressions early

Conclusion: The Never-Ending Quest for Speed

Tuning Linux for 100Gbps microservices is not for the faint of heart. It's a complex dance of hardware capabilities, kernel parameters, and application-level optimizations. But with the techniques we've covered – from NIC offloading to TCP_QUICKACK, from buffer tuning to busy polling – you're now armed with the knowledge to take your high-frequency trading environment to the next level.

Remember, the quest for lower latency and higher throughput is never truly over. Keep experimenting, keep measuring, and above all, keep pushing the boundaries of what's possible. Who knows? Maybe next time we'll be talking about tuning for 400Gbps!

"In the world of high-frequency trading, he who hesitates is lost. But he who tunes his kernel rules the market." - Anonymous Linux Kernel Guru

Now go forth and conquer those packets! And if you've got any killer tuning tips of your own, drop them in the comments. After all, in the cutthroat world of HFT, we're all in this together... until we're not.