Here's the TL;DR for the impatient:

  • eBPF lets you run sandboxed programs in the Linux kernel
  • It provides deep insights into system and application behavior
  • Low overhead means you can use it in production without breaking a sweat
  • It's versatile: from network analysis to security monitoring, eBPF's got your back

eBPF 101: The Basics

Before we dive into the nitty-gritty, let's break down how eBPF works. Think of it as planting tiny, efficient spies throughout your Linux kernel. These spies (eBPF programs) can report on everything from system calls to network packets, all without needing to modify your kernel or reboot your system.

Here's the magic trick:

  1. You write an eBPF program (usually in C)
  2. This program gets compiled to eBPF bytecode
  3. The bytecode is verified for safety (no kernel panics, thank you very much)
  4. It's then JIT-compiled and injected into the kernel
  5. Your program runs when specific events occur, collecting data or modifying behavior

It's like having a team of highly trained, microscopic ninjas ready to report on any aspect of your system at a moment's notice.

Setting Up Your eBPF Lab

Ready to get your hands dirty? Let's set up an eBPF playground. First, you'll need a relatively recent Linux kernel (4.4+, but newer is better). Then, let's install some tools:


sudo apt-get update
sudo apt-get install -y bpfcc-tools linux-headers-$(uname -r)

This installs BCC (BPF Compiler Collection), which gives us a nice Python interface to work with eBPF. Now, let's write our first eBPF program:


from bcc import BPF

# Define the eBPF program
prog = """
int hello(void *ctx) {
    bpf_trace_printk("Hello, eBPF World!\\n");
    return 0;
}
"""

# Load the eBPF program
b = BPF(text=prog)

# Attach it to a kernel function
b.attach_kprobe(event="sys_clone", fn_name="hello")

# Print the output
print("eBPF program loaded. Tracing sys_clone()... Ctrl+C to exit.")
b.trace_print()

Save this as hello_ebpf.py and run it with sudo python3 hello_ebpf.py. Congratulations! You've just created an eBPF program that says hello every time a new process is created.

Process Monitoring: Who's Eating My CPU?

Now that we've dipped our toes in, let's dive deeper. One common use case for eBPF is monitoring process behavior. Let's create a program that tracks CPU usage per process:


from bcc import BPF
from time import sleep

# eBPF program
prog = """
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>

struct key_t {
    u32 pid;
    u64 cpu;
};

BPF_HASH(stats, struct key_t);

int on_switch(struct pt_regs *ctx, struct task_struct *prev) {
    struct key_t key = {};
    u64 delta, *time;

    key.pid = prev->pid;
    key.cpu = bpf_get_smp_processor_id();

    time = stats.lookup(&key);
    if (time) {
        delta = bpf_ktime_get_ns() - *time;
        stats.increment(key, delta);
    }

    key.pid = bpf_get_current_pid_tgid() >> 32;
    stats.update(&key, &bpf_ktime_get_ns());

    return 0;
}
"""

b = BPF(text=prog)
b.attach_kprobe(event="finish_task_switch", fn_name="on_switch")

print("Monitoring CPU usage... Ctrl+C to exit")
while True:
    sleep(1)
    print("\nTop 5 CPU-hungry processes:")
    b["stats"].print_log2_hist("CPU Usage", "pid", section_print_fn=lambda k, v: f"PID {k.pid:<6} CPU {k.cpu:<3}")

This script attaches to the finish_task_switch function, allowing us to track how long each process spends on CPU. Run it, and watch as it reveals which processes are hogging your precious CPU cycles!

Network Sleuthing with eBPF

eBPF isn't just for process monitoring; it's also a powerful tool for network analysis. Let's create a simple program to track network connections:


from bcc import BPF

# eBPF program
prog = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bcc/proto.h>

BPF_HASH(ipv4_sends, u32, u64);

int trace_ip_send(struct pt_regs *ctx, struct sock *sk) {
    u32 dst = sk->__sk_common.skc_daddr;
    ipv4_sends.increment(dst);
    return 0;
}
"""

b = BPF(text=prog)
b.attach_kprobe(event="ip_send_skb", fn_name="trace_ip_send")

print("Tracing IP sends... Ctrl+C to exit")
try:
    while True:
        b.perf_buffer_poll()
except KeyboardInterrupt:
    print("\nTop IP destinations:")
    b["ipv4_sends"].print_log2_hist("Packets", "IP")

This script tracks outgoing IPv4 connections, giving you a bird's-eye view of where your system is sending packets. It's like having your own mini Wireshark built right into the kernel!

Integrating eBPF with Prometheus and Grafana

Now that we're collecting all this juicy data, let's visualize it! We'll use Prometheus to scrape our eBPF metrics and Grafana to create beautiful dashboards.

First, let's modify our CPU monitoring script to expose metrics for Prometheus:


from bcc import BPF
from prometheus_client import start_http_server, Gauge
import time

# ... (previous eBPF program code here)

# Prometheus metrics
PROCESS_CPU_USAGE = Gauge('process_cpu_usage', 'CPU usage by process', ['pid', 'cpu'])

def update_metrics():
    for k, v in b["stats"].items():
        PROCESS_CPU_USAGE.labels(pid=k.pid, cpu=k.cpu).set(v.value)

if __name__ == '__main__':
    b = BPF(text=prog)
    b.attach_kprobe(event="finish_task_switch", fn_name="on_switch")

    # Start Prometheus HTTP server
    start_http_server(8000)

    print("Monitoring CPU usage... Metrics available at :8000")
    while True:
        time.sleep(1)
        update_metrics()

Now, configure Prometheus to scrape these metrics and set up a Grafana dashboard to visualize them. You'll have a real-time view of your system's CPU usage that would make any sysadmin drool!

Security Monitoring: Catching the Bad Guys

eBPF isn't just about performance; it's also a powerful tool for security monitoring. Let's create a simple intrusion detection system that watches for suspicious file accesses:


from bcc import BPF

prog = """
#include <uapi/linux/ptrace.h>
#include <linux/fs.h>

BPF_HASH(suspicious_accesses, u32);

int trace_open(struct pt_regs *ctx, const char __user *filename, int flags) {
    char fname[256];
    bpf_probe_read_user_str(fname, sizeof(fname), (void *)filename);
    
    if (strstr(fname, "/etc/shadow") != NULL) {
        u32 pid = bpf_get_current_pid_tgid() >> 32;
        suspicious_accesses.increment(pid);
    }
    
    return 0;
}
"""

b = BPF(text=prog)
b.attach_kprobe(event="do_sys_open", fn_name="trace_open")

print("Monitoring for suspicious file accesses... Ctrl+C to exit")
try:
    while True:
        b.perf_buffer_poll()
except KeyboardInterrupt:
    print("\nSuspicious accesses detected:")
    b["suspicious_accesses"].print_log2_hist("Attempts", "PID")

This script monitors attempts to access the /etc/shadow file, which could indicate someone trying to steal password hashes. It's a simple example, but it demonstrates how eBPF can be used for real-time security monitoring.

Performance Tuning: Squeezing Out Every Last Drop

Let's use eBPF to identify and optimize slow disk I/O operations:


from bcc import BPF
import time

prog = """
#include <uapi/linux/ptrace.h>
#include <linux/blkdev.h>

BPF_HISTOGRAM(dist);

int trace_req_start(struct pt_regs *ctx, struct request *req) {
    u64 ts = bpf_ktime_get_ns();
    bpf_map_update_elem(&start, &req, &ts, BPF_ANY);
    return 0;
}

int trace_req_completion(struct pt_regs *ctx, struct request *req) {
    u64 *tsp, delta;
    tsp = bpf_map_lookup_elem(&start, &req);
    if (tsp != 0) {
        delta = bpf_ktime_get_ns() - *tsp;
        dist.increment(bpf_log2l(delta / 1000));
        bpf_map_delete_elem(&start, &req);
    }
    return 0;
}
"""

b = BPF(text=prog)
b.attach_kprobe(event="blk_start_request", fn_name="trace_req_start")
b.attach_kprobe(event="blk_mq_start_request", fn_name="trace_req_start")
b.attach_kprobe(event="blk_account_io_completion", fn_name="trace_req_completion")

print("Tracing block device I/O... Hit Ctrl+C to end.")
try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    print("\nBlock I/O latency histogram:")
    b["dist"].print_log2_hist("usecs")

This script measures the latency of disk I/O operations, helping you identify potential bottlenecks in your storage system. Armed with this information, you can make informed decisions about optimizing your disk layout or upgrading your hardware.

eBPF Tools: Your New Best Friends

We've barely scratched the surface of what's possible with eBPF. Here are some powerful tools you should know about:

  • bpftrace: A high-level tracing language for eBPF
  • bcc: The BPF Compiler Collection, which we've been using in our examples
  • cilium: Network security and visibility using eBPF
  • falco: Runtime security monitoring powered by eBPF
  • pixie: Auto-instrumentation and monitoring for Kubernetes

Each of these tools leverages eBPF in unique ways, providing powerful capabilities for monitoring, security, and performance optimization.

Wrapping Up: The eBPF Revolution

eBPF is revolutionizing how we monitor and optimize Linux systems. It gives us unprecedented visibility into system behavior with minimal overhead, allowing us to debug, secure, and optimize our systems in ways that were previously impossible or impractical.

As we've seen, eBPF can be used for:

  • Deep system introspection
  • Performance monitoring and optimization
  • Network analysis and security
  • Custom monitoring solutions

The best part? This is just the beginning. As eBPF continues to evolve, we can expect even more powerful tools and techniques to emerge.

So, the next time you find yourself staring at a mysterious performance issue or scratching your head over an elusive bug, remember: eBPF is your secret weapon. Happy monitoring, and may your systems always run smoothly!

"With great power comes great responsibility." While eBPF gives you superpowers, remember to use them wisely. Always test your eBPF programs thoroughly and be mindful of their impact on system performance.

Now go forth and conquer those Linux systems with your newfound eBPF skills!