Extended Berkeley Packet Filters (eBPF) is probably the most exciting additions to the Linux Kernel in recent times. To put it simply, eBPF is designed as a very nice middle-ground between native kernel changes, which have traditionally been very slow (arguably rightfully) to come into mainline, and kernel modules, which have become a popular footgun for anyone working at the kernel level.
My introduction to eBPF was through Pyroscope, which uses eBPF to provide continuous profiling (And anyone who's worked with code long enough knows the pain that it takes to profile a system running in Prod WITHOUT having a significant impact on performance), and over time, I've been playing around with more tools in the eBPF space, whether it's on the networking front (a la Caretta), or on the security/observability front (a la Cilium / Falco), to wilder shots like load balancing (a la Katran). But while I've grokked the "what" and "why" of eBPF, the "how" was something that was always a mystery to me.
So I dug a little to see how you can write your own eBPF-enabled program, and it turned out to be much easier than I assumed it would be... So let's see how we do it!
For this sample app, we'll write a simple Python program that will act as the userspace application and have it communicate with an eBPF program written in C (which the eBPF compiler will compile to bytecode and run at the kernel level). So let's get started.
First, let's install the bpfcc-tools and linux-headers package, which has what we need(*)
$ sudo apt-get install bpfcc-tools linux-headers-$(uname -r)
And now we can write our code:
$ cat hello_ebpf.py
from bcc import BPF
ebpf_program = r"""
int process_start(void *ctx) {
bpf_trace_printk("A new process was started!");
return 0;
}
"""
prog = BPF(text=ebpf_program)
execve_syscall = prog.get_syscall_fnname("execve")
prog.attach_kprobe(event=execve_syscall, fn_name="process_start")
prog.trace_print()
And that's it! Our eBPF hello-worldis now ready for action. To run it, simply run the file in the terminal
$ sudo python3 hello_ebpf.py
# Depending on your package version, you may see some warnings here
In another terminal, you can kick off some processes by running commands like ls, echo, cat etc. and every time a new process starts (which internally uses the execve syscall), you'll see a message printed in the running python application.
b' bash-3713 [000] d...1 12276.629282: bpf_trace_printk: A new process was started!'
b' bash-3714 [000] d...1 12278.088251: bpf_trace_printk: A new process was started!'
b' python3-3715 [000] d...1 12278.244559: bpf_trace_printk: A new process was started!'
So, how does this work? Well, the code should be quite self-explanatory, but the important bit is the prog.attach_kprobe(event=execve_syscall, fn_name="process_start") line. What this does is create and attach a kprobe to each call to execve and runs a hook defined in process_start method, which is what is compiled to eBPF bytecode.
You can probably appreciate how this makes writing applications that can access kernel level data so much easier, and allow access at an incredibly detailed level. Want to know when a file is accessed? eBPF allows that. Want to know when a process talks to another? No problemo. Want to know when a network socket request is made? Easy, peasy!(^) eBPF's relatively straightforward interface means that even with so much fine-grained access to syscalls, picking, filtering, slicing, and dicing kernel calls is straightforward, without needing as much low-level kernel knowledge. Best of all, since eBPF is compiled to bytecode by the eBPF compiler and executed within the kernel rather than in userspace, the performance impact of these listeners is EXTREMELY low. This allows real-world applications like continuous profiling, network security, and load balancing to be driven through eBPF without worrying too much about real-world performance.
I've been playing with eBPF over the last year or so and would definitely recommend folks to look into this incredible kernel technology. Who knows, it may just be the system-level observability "silver bullet" you've been looking for.
---
(*) I'm running this on Ubuntu 22.04. Instructions may be a bit different depending on your distro. But as long as you have a fairly modern kernel version (4.1+) you should be able to get this up and running.
(^) If there's interest, I may just make this a multi-part series diving deeper into how to write these applications as well.
