Ebpf on Peter McConnell :: Ponderings from a Linux Systems engineer

Snooping on libpam (openssh auth, passwd) with Golang and eBPF

Sun, 18 Feb 2024 11:40:02 +0000

In the vast and complex landscape of software security, safeguarding sensitive information remains a paramount concern for developers and security professionals alike. Among the myriad of challenges, securely managing and protecting credentials during authentication processes stands out as a critical vulnerability point. Traditional security measures often fall short in providing real-time insights into how credentials are handled and potentially exposed within applications, especially those relying on widely used authentication frameworks like PAM (Pluggable Authentication Modules).

I work for a runtime security platform so this topic, BPF, Golang and Linux are some that I find the most interesting. Tangentally I was asked to give a talk at a Golang meetup, so figured I’d create a mini project to showcase some of this tech may be a good way to get others excited about these topics. Hence this repo and article.

The source code for this project can be seen here: https://github.com/peter-mcconnell/whispers.

A high-level introduction to some of the tech we’ll be playing with

Two pivotal technologies at the heart of our work here are uprobes (user space probes) and eBPF (Extended Berkeley Packet Filter).

Uprobes: User Space Probes

Uprobes are a dynamic tracing feature of the Linux kernel that allows developers to instrument user space binaries. By attaching to specific locations in the executable code (such as function entry or exit points), uprobes enable the collection of data about the binary’s execution without modifying its code. This makes uprobes an invaluable tool for performance analysis, debugging, and, as with whispers, security monitoring.

When a specified location is executed, the uprobe triggers, and control is passed to a handler function, which can gather information about the execution context, such as function arguments, return values, and the process ID. This capability allows whispers to monitor authentication processes by attaching uprobes to critical functions within libpam, capturing credential data as it’s passed through these functions.

eBPF: Extended Berkeley Packet Filter

eBPF is a revolutionary technology that extends the traditional Berkeley Packet Filter (BPF) with enhanced capabilities, allowing for the safe execution of small programs within the Linux kernel without changing kernel source code or loading kernel modules. eBPF programs are written in a restricted C-like language, compiled into bytecode, and then executed by the kernel’s virtual machine, ensuring safety and efficiency.

eBPF has a wide range of applications, from networking and security to performance monitoring and debugging. It interacts with the kernel and user space through maps (data structures for storing state) and program types (defining what an eBPF program can do).

For our project, eBPF is used to implement uprobes for capturing authentication data. The eBPF program is attached to the target function in the user space binary, and upon function execution, the eBPF program is triggered, capturing relevant data and passing it back to user space for analysis. This seamless interaction between user and kernel space is what enables whispers to monitor and log credential information efficiently and transparently. The combination of uprobes and eBPF offers a powerful mechanism for introspecting and monitoring system and application behavior in real-time. With uprobes providing the ability to hook into specific points of interest in user space applications and eBPF facilitating the safe execution of custom logic within the kernel, developers can create sophisticated monitoring tools like whispers that are both efficient and minimally invasive.

This innovative approach to system monitoring not only enhances security by exposing potential vulnerabilities and data exposures but also serves as an educational tool, shedding light on the inner workings of system authentication processes and the capabilities of modern Linux kernel technologies.

BPF Maps: Bridging Kernel and User Space

An integral component of eBPF’s powerful capabilities comes from BPF maps, sophisticated data structures designed to store and share data between eBPF programs running in the kernel and applications in user space. These maps play a crucial role in maintaining state, passing information, and facilitating complex data management tasks within eBPF-driven applications like whispers. What Are BPF Maps?

BPF maps are key-value stores that can be accessed by eBPF programs and user space applications. They come in various types, each optimized for specific use cases, such as arrays, hash maps, and ring buffers. The choice of map type depends on the nature of the data being stored and the access patterns required by the eBPF program and the user space application.

Key Features and Benefits:

Efficient Data Sharing: BPF maps provide a high-performance mechanism for sharing data between the kernel and user space, crucial for applications that require real-time processing and analysis.
Statefulness: Unlike traditional stateless packet processing in the kernel, BPF maps enable eBPF programs to maintain state across function calls and packet processing stages, allowing for more sophisticated logic and tracking capabilities.
Versatility: The variety of available map types makes BPF maps suitable for a wide range of applications, from performance monitoring and networking to security and observability tools like whispers.
Safety and Isolation: Access to BPF maps from eBPF programs is rigorously checked by the kernel’s verifier, ensuring that only valid, safe operations are performed, thereby protecting the kernel from potentially malicious or buggy code.

In the context of our project “whispers”, BPF maps serve as the primary mechanism for transferring captured credential data from the eBPF program (attached to libpam functions via uprobes) to the user space component of the tool. For instance, ring buffers are utilized to efficiently pass event data (such as authentication attempts) from kernel to user space, where it can be processed, analyzed, and logged. This enables whispers to monitor and report on authentication processes with minimal overhead and without requiring direct access to the monitored application’s memory or internal data structures.

Knowing what to look for

We know that we want to snoop for creds on libpam, as used by OpenSSH. In order to “snoop” on libpam we need to know which functions and structures of this library we should snoop on. We also know that we wish to attach uprobes, so are relying on the available symbols of libpam - that seems a reasonable place to start.

Creating a debug environment

When setting off on a new adventure I like to create a little investigation environment that allows me to poke at things, break them etc. Often this results in a Dockerfile and this case is no different:

FROM ubuntu:22.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update -yq && \
		apt-get install -yq openssh-server && \
		# some debug utilities, to aide exploration
		apt-get install -yq binutils bpftrace systemtap systemtap-sdt-dev linux-headers-$(uname -r) vim && \
		mkdir -p /var/run/sshd && \
		echo 'root:pass' | chpasswd && \
		sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \
		echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

This not only installs and configures openssh, but also includes some toys for debugging. Using this Dockerfile I docker build -t=debug . and docker run --privileged -p 2222:22 --rm --name debug -d debug to get it going in the background. I then docker exec -ti debug bash to poke around. We’re running --privileged so that we can test attaching probes and running -p 2222:22 so we can attempt to ssh into the container from host.

The screenshot above shows the simple process of hunting down where libpam is using sshd as a starting point. I could have just done find / -name "libpam*" but I wanted to be extra sure it was linked from openssh, which is what I’m intending on using for my tests.

Now that I’ve identified the location for libpam, and that it is indeed linked from sshd, I can proceed to identify symbols which may be of interest. To do this I simply ask readelf to dump the symbols and grep the output for auth which is a term I’d expect to see in the symbol name for a function that handles authentication.

root@664e76f33de2:/# readelf -Ws /lib/x86_64-linux-gnu/libpam.so.0 | grep auth
    82: 00000000000088e0   699 FUNC    GLOBAL DEFAULT   15 pam_get_authtok_verify@@LIBPAM_EXTENSION_1.1.1
    93: 00000000000088b0    12 FUNC    GLOBAL DEFAULT   15 pam_get_authtok@@LIBPAM_EXTENSION_1.1
   112: 00000000000088c0    26 FUNC    GLOBAL DEFAULT   15 pam_get_authtok_noverify@@LIBPAM_EXTENSION_1.1.1
   116: 0000000000009ab0   371 FUNC    GLOBAL DEFAULT   15 pam_chauthtok@@LIBPAM_1.0
   118: 0000000000009940   259 FUNC    GLOBAL DEFAULT   15 pam_authenticate@@LIBPAM_1.0

These all seem reasonably interesting. To ensure these are the right things to hook onto I test them out using `bpftrace -e ‘uprobe:/lib/x86_64-linux-gnu/libpam.so.0:pam_get_authtok { printf(“pam_get_authtok called\n”); }’

With bpftrace running and the uprobe trace attached I then ssh into the container using ssh root@localhost -p 2222:22 from another terminal and can see the trace is made when authentication is handled. Seems like we’re on the right track.

Now I have something to look for in the source code so head over to https://github.com/linux-pam/linux-pam/ where after a bit of searching for the symbols I find https://github.com/linux-pam/linux-pam/blob/1e2c6cecf81dcaeea0c2c9d37bc35eea120cd77d/libpam/pam_get_authtok.c#L213 which looks like the function signature we’re after:

int
pam_get_authtok (pam_handle_t *pamh, int item, const char **authtok,
		 const char *prompt)
{
  return pam_get_authtok_internal (pamh, item, authtok, prompt, 0);
}

With our BPF program we’ll intercept the arguments, so first thing to look up is pam_handle_t which after some scouring leads to https://github.com/linux-pam/linux-pam/blob/1e2c6cecf81dcaeea0c2c9d37bc35eea120cd77d/libpam/pam_private.h#L154:


struct pam_handle {
    char *authtok;
    unsigned caller_is;
    struct pam_conv *pam_conversation;
    char *oldauthtok;
    char *prompt;                /* for use by pam_get_user() */
    char *service_name;
    char *user;
    char *rhost;
    char *ruser;
    char *tty;
    char *xdisplay;
    char *authtok_type;          /* PAM_AUTHTOK_TYPE */
    struct pam_data *data;
    struct pam_environ *env;      /* structure to maintain environment list */
    struct _pam_fail_delay fail_delay;   /* helper function for easy delays */
    struct pam_xauth_data xauth;        /* auth info for X display */
    struct service handlers;
    struct _pam_former_state former;  /* library state - support for
					 event driven applications */
    const char *mod_name;	/* Name of the module currently executed */
    int mod_argc;               /* Number of module arguments */
    char **mod_argv;            /* module arguments */
    int choice;			/* Which function we call from the module */

#ifdef HAVE_LIBAUDIT
    int audit_state;             /* keep track of reported audit messages */
#endif
    int authtok_verified;
    char *confdir;
};

This has some interesting fields like authtok and user. At this point it feels like we have a reasonable direction for what to hook into.

Developing the eBPF Program for whispers

This section provides a high level overview at the development of the eBPF program for whispers, a tool designed to monitor and capture credentials handled by libpam. By leveraging eBPF, whispers taps into kernel functions related to authentication processes, offering unprecedented insights into credential management and security vulnerabilities. You can check out all of the source code here: https://github.com/peter-mcconnell/whispers.

Dependencies

Before diving into eBPF program development, ensure your development environment is properly set up with the following tools and libraries:

LLVM & Clang: For compiling eBPF programs.
Linux Kernel Headers: Required for eBPF program compilation to access kernel APIs.
libbpf: A library for loading and interacting with eBPF programs and maps.
Go toolchain: For the Go-based components of whispers.

Developing the eBPF Program

The eBPF program for whispers is designed to attach to the pam_get_authtok function via a uretprobe, capturing credentials as they’re processed.

uretprobe.c: Contains the eBPF code that defines the data structures and logic for capturing and processing credential information from pam\_get\_authtok.

When trying to look up the structure of pam_get_authtok I encountered https://github.com/citronneur/pamspy which is where I lifted most of this code from, so credit to them, but you can see this just matches the structure we found in the libpam source code mentioned above:

https://github.com/peter-mcconnell/whispers/blob/main/bpf/uretprobe.c

...

// we need the pam_handle struct so that we can read the data in our BPF program
typedef struct pam_handle
{
  char *authtok;
  unsigned caller_is;
  void *pam_conversation;
  char *oldauthtok;
  char *prompt;
  char *service_name;
  char *user;
  char *rhost;
  char *ruser;
  char *tty;
  char *xdisplay;
  char *authtok_type;
  void *data;
  void *env;
} pam_handle_t;

// we're using a ringbuffer datastructure to share data back to our userspace program
struct
{
  __uint(type, BPF_MAP_TYPE_RINGBUF);
  __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

// and we will use this struct to store the items for our bpf map
typedef struct _event_t {
    int  pid;
    char comm[16];
    char username[80];
    char password[80];
} event_t;
...

SEC("uretprobe/pam_get_authtok")
int trace_pam_get_authtok(struct pt_regs *ctx)
{
  if (!PT_REGS_PARM1(ctx))
    return 0;

  pam_handle_t* phandle = (pam_handle_t*)PT_REGS_PARM1(ctx);

  u32 pid = bpf_get_current_pid_tgid() >> 32;

  u64 password_addr = 0;
  bpf_probe_read(&password_addr, sizeof(password_addr), &phandle->authtok);

  u64 username_addr = 0;
  bpf_probe_read(&username_addr, sizeof(username_addr), &phandle->user);

  event_t *e;
  e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
  if (e)
  {
    e->pid = pid;
    bpf_probe_read(&e->password, sizeof(e->password), (void *)password_addr);
    bpf_probe_read(&e->username, sizeof(e->username), (void *)username_addr);
    bpf_get_current_comm(&e->comm, sizeof(e->comm));
    bpf_ringbuf_submit(e, 0);
  }
  return 0;
};

To handle compiling this code, given we’re going to be using Golang we’ll define a go generate later which will point to a bash script which will handle compilation. For now though we can create the gen.sh which will handle the compilation:

#!/usr/bin/env bash

set -e
set -x

# Dynamically find kernel headers. Adjust this line according to your needs.
KERNEL_HEADERS=$(find /usr/src -name "linux-headers-$(uname -r)" -type d | head -n 1)

# Include directory for bpf_helpers.h. This might need adjustments.
BPF_HELPERS_DIR="${KERNEL_HEADERS}/tools/bpf/resolve_btfids/libbpf/include/"

# Run bpf2go with dynamic include paths
go run github.com/cilium/ebpf/cmd/bpf2go -target amd64 bpf \
    ../../bpf/uretprobe.c -- \
    -I"${KERNEL_HEADERS}" \
    -I"${BPF_HELPERS_DIR}" \
    -I../../bpf/headers

This will be invoked by go build whenever it encounters a // go generate ../path/to/gen.sh directive. go generate is a command provided by the Go toolchain that automates the generation of code before the build process. It scans Go source files for special comments that specify commands to be run at generate time. These commands can invoke any process but are commonly used to generate Go code, making it a perfect fit for integrating eBPF programs into Go applications.

Golang code

This section delves into the Go component of whispers, which serves as the user-space counterpart to the eBPF program. It outlines how to leverage Go to load the eBPF program, attach probes, read from eBPF maps, and process the captured credential data for monitoring and analysis purposes.

Ensure your Go environment is set up with the necessary dependencies:

Go (1.21 or newer): Ensure the Go toolchain is installed.

The Go codebase for whispers is structured as follows:

cmd/: Contains the CLI interface for whispers.
pkg/config: Defines configuration structures.
pkg/whispers: Implements the core functionality for loading and interacting with the eBPF program.

Dependencies are defined in go.mod and can be pulled with go mod tidy.

This article will only touch on the most interesting parts of the code. The source can be read at https://github.com/peter-mcconnell/whispers/blob/main/pkg/whispers/whispers.go.

We’re using cilium ebpf to handle loading our BPF program and interacting with the BPF maps. In the following, bpfObjects and loadBpfObjects are defined in the auto-generated file bpf_x86_bpfel.go which is generated during go build due to the // go:generate ... directive which triggers gen.sh:

func Listen(ctx context.Context, cfg *config.Config) error {
    objs := bpfObjects{}
    if err := loadBpfObjects(&objs, nil); err != nil {
        return err
    }
    defer objs.Close()

    ...
}

The above snippet demonstrates loading the eBPF program and ensuring it is correctly cleaned up upon termination.

whispers attaches uretprobes to target functions (e.g., pam_get_authtok) also using the cilium/ebpf library:

ex, err := link.OpenExecutable(cfg.BinPath)
if err != nil {
    return err
}
up, err := ex.Uretprobe(cfg.Symbol, objs.TracePamGetAuthtok, nil)
if err != nil {
    return err
}
defer up.Close()
...

The Go code reads from the BPF map (a ring buffer) to read events captured by the eBPF program:

rb, err := ringbuf.NewReader(objs.Rb)
if err != nil {
    log.Printf("failed to open ring buffer reader: %v", err)
}
defer rb.Close()

go func() {
    for {
        record, err := rb.Read()
        ...
        event := parseEventData(record.RawSample)
        ...
    }
}()

We’ll need to handle reading the raw data from the BPF map and converting it into something we can easily interact with in Golang:

...
// this needs to perfectly match the structure of our BPF map items (defined in our .h file as _event_t)
type eventT struct {
	Pid      int32
	Comm     [16]byte
	Username [80]byte
	Password [80]byte
}
...
func byteArrayToString(b []byte) string {
	n := -1
	for i, v := range b {
		if v == 0 {
			n = i
			break
		}
	}
	if n == -1 {
		n = len(b)
	}
	return string(b[:n])
}

func parseEventData(data []byte) *eventT {
	var event eventT

	if len(data) >= int(unsafe.Sizeof(eventT{})) {
		if err := binary.Read(bytes.NewReader(data), binary.LittleEndian, &event); err == nil {
			return &event
		}
	}

	return nil
}
...

After reading events from the ring buffer, whispers processes and logs the captured credentials:

log.Printf("Event: PID: %d, Comm: %s, Username: %s, Password: %s",
    event.Pid,
    byteArrayToString(event.Comm[:]),
    byteArrayToString(event.Username[:]),
    byteArrayToString(event.Password[:]))

Seeing it in action

Included with the repository is a Dockerfile that not only builds whisper but also contains an sshd server ready to go, so that you can play with it immediately. The following is a demonstration of that:

Improvements

This project was designed to be very simple - specifically I’m aiming to give an overview of it during a tech talk. As such, it isn’t intended to be feature-complete. Namely this project results in a binary whereby you provide the -binPath=/path/to/libpam, however we can automate this attachment with more eBPF.

This project has used uprobes to trace arguments for a given user-space application. However we can automate the auto-attachment of these probes using kprobes. With kprobes we could filter execve and fork for comms that we deem to be interesting (e.g. sshd, passwd) and when we get a match, we could run ldd against the /proc/<pid>/exe to see if we can see libpam and attach that way. There are some complexities to consider here such as the namespace for a given process may be different (e.g. for containers). We’d likely also want to look at existing processes and attach probes to those also. However, these are all trivial problems to solve. The result being that you could run a single whispers with no targets, that would automatically attach our uprobes whenever a suitable process was started.

Building an XDP eBPF Program with C and Golang: A Step-by-Step Guide

Wed, 17 May 2023 15:31:43 +0000

Introduction

In today’s highly connected and data-driven world, network performance is crucial for ensuring efficient communication and optimal user experience. XDP (eXpress Data Path) and eBPF (extended Berkeley Packet Filter) have emerged as powerful technologies that enable high-performance packet processing and network optimization. In this step-by-step guide, we will explore the process of building an XDP eBPF program using C and Golang. XDP allows for early packet interception at the network interface driver level, while eBPF provides a flexible and efficient execution environment for custom packet processing logic. Together, they offer an unprecedented level of control and performance in networking applications. Our project, named “dilih” (drop it like it’s hot), demonstrates how to build a simple chaos engineering tool that arbitrarily drops packets on a given network interface, which can be a useful tool to allow application developers to understand how their products behave when the network isn’t. Through this guide, you will gain a basic understanding of XDP, eBPF, and their practical applications in network manipulation.

note: you can find the entire codebase for this article here: https://github.com/peter-mcconnell/dilih

Project Overview

The project aims to build an XDP (eXpress Data Path) eBPF (extended Berkeley Packet Filter) program using C and Golang. Named “dilih” (drop it like it’s hot, as it drops packets like … they’re hot?), the program serves as a simple chaos engineering tool that randomly drops around 50% of packets on a given network interface. This project demonstrates the power and flexibility of XDP and eBPF in controlling packet processing at high speed, making it an ideal starting point for understanding these technologies.

The XDP eBPF program, implemented in C, hooks into the Linux kernel’s networking stack at an early stage to intercept packets and decide their fate. Using a simple randomization mechanism the program selectively drops packets allowing for controlled chaos in network traffic. Additionally, the program utilizes eBPF’s perf event mechanism to gather statistics and measure the processing time for dropped and passed packets.

The accompanying Golang application interacts with the XDP eBPF program, providing a user-friendly interface to monitor the packet drop behavior and visualize performance statistics. It leverages eBPF maps to extract and aggregate the collected data from kernel space, allowing users to gain insights into the impact of dropped packets and the efficiency of packet processing.

Setting Up the Development Environment

To get started with building the XDP eBPF program with C and Golang, you need to set up your development environment. Follow these steps to ensure that you have all the necessary tools and dependencies in place:

Install Development Tools

First, ensure that you have the required development tools installed on your system. This includes packages like clang, llvm, and bpftool. You can install these tools using the package manager available on your Linux distribution however I would recommend investing a little time to build these tools from source as it will give you greater control over the flags and features built into these tools.

If you are curious about my exact LLVM / Clang setup, I use the following ansible tasks for my configuration:

https://github.com/peter-mcconnell/.dotfiles/blob/master/tasks/llvm.yaml https://github.com/peter-mcconnell/.dotfiles/blob/master/tasks/debugtools.yaml https://github.com/peter-mcconnell/.dotfiles/blob/master/tasks/docker.yaml
Install Golang

Next, you need to install Golang, which is the programming language used for the accompanying Golang application. Visit the official Golang website at https://golang.org and follow the installation instructions specific to your operating system. Once installed, make sure the go command is accessible from the command line by adding the appropriate binary directory to your system’s PATH.

If you are curious about my exact Golang setup, I use the following ansible task for my configuration:

https://github.com/peter-mcconnell/.dotfiles/blob/master/tasks/golang.yaml
Install Project Dependencies

go dependencies

Navigate to the project’s root directory and install the required Golang dependencies by running the following command:
```
go mod download
```
This command will fetch and install the necessary Golang packages defined in the project’s go.mod file.

libbpf

We are going to use libbpf in our C code. In the dilih repo we added this as a git submodule but you can choose to manage it elsewhere if you like. When we build our C program later we’ll include libbpf with -I../libbpf/src
(Optional) IDE configuration

Whatever your editor of choice should be, invest some time in making sure it is set up for C and Golang. Particularly for autocomplete, linting, symbol detection etc. This will make your life much easier.

If you are curious about my exact setup, I use the following repo to install neovim, configure my LSP and setup everything else I need for development:

https://github.com/peter-mcconnell/.dotfiles/

Writing the XDP eBPF Program in C

The XDP (eXpress Data Path) program is implemented using the eBPF (extended Berkeley Packet Filter) framework in C, with some help from libbpf. It allows us to intercept packets at an early stage in the Linux kernel’s networking stack and perform custom packet processing logic. In this section, we will walk through the steps to write the XDP eBPF program in C.

Understanding the Program Logic

Before diving into the code, let’s understand the logic of our XDP program. The goal is to randomly drop ~50% of packets on a given network interface. We will use a randomization mechanism to decide whether to drop or pass each packet (“is random number even?”). The program will also collect statistics and measure the processing time for dropped and passed packets using eBPF’s perf event mechanism. Our BPF program runs in kernel space but we’ll want to expose data to userspace, so we’ll use BPF maps to expose data to our Go program.
Creating the Program Source File

Start by creating a new file called dilih_kern.c in a ./bpf/ directory in your project. This file will contain our XDP eBPF program logic. Open the file in your favorite text editor.
Defining the required headers and structures

To begin, include the necessary headers and define the required structures for our XDP program. We need bpf.h and bpf_helpers.h which provide useful structures and helper functions.
```
#include <linux/bpf.h>
#include <bpf_helpers.h>
```
Defining Data Structures and Maps

Next, define the necessary data structures and maps that our XDP program will utilize. We will use a struct to represent the perf event data, and a BPF_MAP_TYPE_PERF_EVENT_ARRAY map to store the perf events. Define the following structures and maps:
```
struct perf_trace_event {
    __u64 timestamp;
    __u32 processing_time_ns;
    __u8 type;
};

#define TYPE_ENTER 1
#define TYPE_DROP 2
#define TYPE_PASS 3

struct {
    __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
    __uint(key_size, sizeof(int));
    __uint(value_size, sizeof(struct perf_trace_event));
    __uint(max_entries, 1024);
} output_map SEC(".maps");
```
The output_map map will be used to store the perf events generated by our XDP program. The TYPE_* definitions will make our code more readable later.
Implementing the XDP Program Function

Now, it’s time to implement the XDP program function itself. Begin by declaring the XDP function with the appropriate signature:
```
SEC("xdp")
int xdp_dilih(struct xdp_md *ctx)
{
    // Add program logic here ... detailed in the next step
}
```
The xdp_dilih function will serve as our XDP eBPF program entry point. It will be called for every incoming packet.
Handling Perf Events and Collecting Data

Inside the xdp_dilih function, we can handle perf events to collect data and measure processing time. We have already defined the output_map to store these events. Use the bpf_perf_event_output helper function to emit perf events to the map.
```
struct perf_trace_event e = {};

// Perf event for entering xdp program
e.timestamp = bpf_ktime_get_ns();
e.type = TYPE_ENTER;
e.processing_time_ns = 0;
bpf_perf_event_output(ctx, &output_map, BPF_F_CURRENT_CPU, &e, sizeof(e));

// Packet dropping logic
if (bpf_get_prandom_u32() % 2 == 0) {
    // Perf event for dropping packet
    e.type = TYPE_DROP;
    __u64 ts = bpf_ktime_get_ns();
    e.processing_time_ns = ts - e.timestamp;
    e.timestamp = ts;
    bpf_perf_event_output(ctx, &output_map, BPF_F_CURRENT_CPU, &e, sizeof(e));
    return XDP_DROP;
}

// Perf event for passing packet
e.type = TYPE_PASS;
__u64 ts = bpf_ktime_get_ns();
e.processing_time_ns = ts - e.timestamp;
e.timestamp = ts;
bpf_perf_event_output(ctx, &output_map, BPF_F_CURRENT_CPU, &e, sizeof(e));

return XDP_PASS;
```
In this section of the code, we handle the perf events to collect data and measure the processing time of dropped and passed packets. We first emit a perf event when entering the XDP program (type 1). Then, we use a randomization mechanism to decide whether to drop or pass the packet. If the packet is dropped, we emit a perf event with type 2 and return XDP_DROP. If the packet is passed, we emit a perf event with type 3 and return XDP_PASS.

The bpf_ktime_get_ns() function is used to measure the timestamp (nanoseconds since system boot, excluding suspend time) and processing time of the packet. The bpf_get_prandom_u32() function generates a random value that helps in deciding whether to drop or pass the packet - the “is this random number even?” part.

Additionally, we use bpf_printk() to print debug messages that can be accessed through the kernel’s trace buffer.

That concludes the implementation of the XDP eBPF program in C. This program will selectively drop packets based on a randomization mechanism and emit perf events for collecting data and measuring processing time.

Compiling and Loading the XDP eBPF Program

Once we have written the XDP eBPF program in C, the next step is to compile it and load it into the kernel. In this section, we will walk through the steps to compile and load the XDP eBPF program.

Compiling the XDP Program

To compile the XDP program, we will use the LLVM Clang compiler with the appropriate flags. Open a terminal and navigate to the bpf directory where the dilih_kern.c file is located. Then, run the following command:

clang -S \
    -g \
    -target bpf \
    -I../libbpf/src\
    -Wall \
    -Werror \
    -O2 -emit-llvm -c -o dilih_kern.ll dilih_kern.c

Let me explain the flags:

-S Emit an intermediate representation (IR) assembly code file instead of generating object code. This step is used to generate LLVM IR code.
-g Include BTF information
-target bpf Specify the target architecture as “bpf” (Berkeley Packet Filter), indicating that the code being compiled is intended to run on eBPF.
-I../libbpf/src Add the path ../libbpf/src to the include search paths. This allows the compiler to find the necessary header files (bpf helper files) from the libbpf library.``Wall` Enable all compiler warning messages.
-Werror Treat all warnings as errors, causing the compilation process to fail if any warnings are encountered.
-O2 Apply optimization level 2 to the generated code. This level of optimization focuses on improving performance without sacrificing code size. This is actually a requirement for some BPF usecases, though I’m struggling to recall right now what they are. TODO
-emit-llvm Instruct the compiler to emit LLVM IR code as the output.
-c Compile the input source file without linking, producing an object file.
-o dilih_kern.ll: Specify the output file name for the generated LLVM IR code as dilih_kern.ll.

Now we use the llc command is used to further process the LLVM IR code and generate the final object file:

llc -march=bpf -filetype=obj -O2 -o dilih_kern.o dilih_kern.ll

Let me explain the flags:

-march=bpf Specify the target architecture as “bpf” for the code generation stage.
-filetype=obj Specify the desired output file type as an object file.
-O2 Apply optimization level 2 to the generated code during the code generation stage.
-o Specify the output file name for the generated object code as dilih_kern.o.

This command compiles the dilih_kern.c file into a BPF object file named dilih_kern.o. The -target bpf flag specifies the target architecture as BPF, and the -O2 flag enables optimization.

Loading the XDP Program

To load the XDP program into the kernel, we will use the bpftool command-line utility. Ensure that you have the bpftool utility installed on your system. If it’s not already installed, you can typically install it using your distribution’s package manager.

In the terminal, run the following command to load the XDP program:

shell sudo bpftool prog load dilih\_kern.o /sys/fs/bpf/dilih

This command loads the dilih_kern.o object file and pins it into the /sys/fs/bpf/dilih location. Adjust the path as necessary based on your system configuration. The bpftool utility will handle the loading process and verify the program’s validity.

Attaching the XDP Program

After loading the XDP program, we need to attach it to a network interface to start intercepting packets. To attach the XDP program, run the following command:

sudo bpftool net attach xdp pinned /sys/fs/bpf/dilih dev <interface>
# you can get <interface> by running `ip link'

Replace with the name of the network interface you want to attach the XDP program to. For example, eth0. This command attaches the XDP program to the specified interface, enabling it to intercept incoming packets.

Makefile

For convenience, lets throw some of what we learned above into a Makefile at ./bpf/Makefile. I’ll not go into depths about how Makefiles work in this article but will summarise the functionality after the code snippet:

TARGET = dilih
BPF_TARGET = ${TARGET:=_kern}
BPF_C = ${BPF_TARGET:=.c}
BPF_OBJ = ${BPF_C:.c=.o}

BPF_PINNED_PATH := /sys/fs/bpf/$(TARGET)
XDP_NAME := dilih
DEV := ens160

xdp: $(BPF_OBJ)
        -bpftool net detach xdpgeneric dev $(DEV)
        rm -f $(BPF_PINNED_PATH)
        bpftool prog load $(BPF_OBJ) $(BPF_PINNED_PATH)
        bpftool net attach xdpgeneric pinned $(BPF_PINNED_PATH) dev $(DEV)

$(BPF_OBJ): %.o: %.c
        clang -S \
                -g \
                -target bpf \
          -I../libbpf/src\
                -Wall \
                -Werror \
                -O2 -emit-llvm -c -o ${@:.o=.ll} $<
        llc -march=bpf -filetype=obj -O2 -o $@ ${@:.o=.ll}

clean:
        -bpftool net detach xdpgeneric dev $(DEV)
        sudo rm -f $(BPF_PINNED_PATH)
        rm -f $(BPF_OBJ)
        rm -f ${BPF_OBJ:.o=.ll}

With this file in place and make installed you can run something like DEV=eth0 make to compile and load the eBPF program and DEV=eth0 make clean to remove the files and unload the eBPF program.

Congratulations! You have successfully compiled and loaded the XDP eBPF program into the kernel and attached it to a network interface. The program is now ready to intercept and process packets based on your defined logic.

Please note that the compilation and loading process may vary slightly depending on your system configuration and specific requirements. Make sure to adjust the commands accordingly and refer to the documentation of the tools and utilities used.

Writing the Golang Application

In this section, we will write a Golang application that interacts with the XDP eBPF program and collects metrics. The Golang application will communicate with the loaded XDP program, read the perf events, and display statistics based on the collected data.

Writing the Golang Application Code

Let’s create a new file named main.go and open it in a text editor. This file will contain the code for our Golang application. Copy and paste the following code into main.go:

package main

import (
        "encoding/binary"
        "fmt"
        "net"
        "os"
        "os/signal"
        "syscall"

        "github.com/cilium/ebpf"
        "github.com/cilium/ebpf/link"
        "github.com/cilium/ebpf/perf"
)

const (
        TYPE_ENTER = 1
        TYPE_DROP  = 2
        TYPE_PASS  = 3
)

type event struct {
        TimeSinceBoot  uint64
        ProcessingTime uint32
        Type           uint8
}

const ringBufferSize = 128 // size of ring buffer used to calculate average processing times
type ringBuffer struct {
        data   [ringBufferSize]uint32
        start  int
        pointer int
        filled bool
}

func (rb *ringBuffer) add(val uint32) {
        if rb.pointer < ringBufferSize {
                rb.pointer++
        } else {
                rb.filled = true
                rb.pointer= 1
        }
        rb.data[rb.pointer-1] = val
}

func (rb *ringBuffer) avg() float32 {
        if rb.pointer == 0 {
                return 0
        }
        sum := uint32(0)
        for _, val := range rb.data {
                sum += uint32(val)
        }
        if rb.filled {
                return float32(sum) / float32(ringBufferSize)
        }
        return float32(sum) / float32(rb.pointer)
}

func main() {
        spec, err := ebpf.LoadCollectionSpec("bpf/dilih_kern.o")
        if err != nil {
                panic(err)
        }

        coll, err := ebpf.NewCollection(spec)
        if err != nil {
                panic(fmt.Sprintf("Failed to create new collection: %v\n", err))
        }
        defer coll.Close()

        prog := coll.Programs["xdp_dilih"]
        if prog == nil {
                panic("No program named 'xdp_dilih' found in collection")
        }

        iface := os.Getenv("INTERFACE")
        if iface == "" {
                panic("No interface specified. Please set the INTERFACE environment variable to the name of the interface to be use")
        }
        iface_idx, err := net.InterfaceByName(iface)
        if err != nil {
                panic(fmt.Sprintf("Failed to get interface %s: %v\n", iface, err))
        }
        opts := link.XDPOptions{
                Program:   prog,
                Interface: iface_idx.Index,
                // Flags is one of XDPAttachFlags (optional).
        }
        lnk, err := link.AttachXDP(opts)
        if err != nil {
                panic(err)
        }
        defer lnk.Close()

        fmt.Println("Successfully loaded and attached BPF program.")

        // handle perf events
        outputMap, ok := coll.Maps["output_map"]
        if !ok {
                panic("No map named 'output_map' found in collection")
        }
        perfEvent, err := perf.NewReader(outputMap, 4096)
        if err != nil {
                panic(fmt.Sprintf("Failed to create perf event reader: %v\n", err))
        }
        defer perfEvent.Close()
        buckets := map[uint8]uint32{
                TYPE_ENTER: 0, // bpf program entered
                TYPE_DROP: 0, // bpf program dropped
                TYPE_PASS: 0, // bpf program passed
        }

        processingTimePassed := &ringBuffer{}
        processingTimeDropped := &ringBuffer{}

        go func() {
                // var event event
                for {
                        record, err := perfEvent.Read()
                        if err != nil {
                                fmt.Println(err)
                                continue
                        }

                        var e event
                        if len(record.RawSample) < 12 {
                                fmt.Println("Invalid sample size")
                                continue
                        }
                        // time since boot in the first 8 bytes
                        e.TimeSinceBoot = binary.LittleEndian.Uint64(record.RawSample[:8])
                        // processing time in the next 4 bytes
                        e.ProcessingTime = binary.LittleEndian.Uint32(record.RawSample[8:12])
                        // type in the last byte
                        e.Type = uint8(record.RawSample[12])
                        buckets[e.Type]++

                        if e.Type == TYPE_ENTER {
                                continue
                        }
                        if e.Type == TYPE_DROP {
                                processingTimeDropped.add(e.ProcessingTime)
                        } else if e.Type == TYPE_PASS {
                                processingTimePassed.add(e.ProcessingTime)
                        }

                        fmt.Print("\033[H\033[2J")
                        fmt.Printf("total: %d. passed: %d. dropped: %d. passed processing time avg (ns): %f. dropped processing time avg (ns): %f\n", buckets[TYPE_ENTER], buckets[TYPE_PASS], buckets[TYPE_DROP], processingTimePassed.avg(), processingTimeDropped.avg())
                }
        }()

        c := make(chan os.Signal, 1)
        signal.Notify(c, os.Interrupt, syscall.SIGTERM)
        <-c
}

You may need to run go mod init && go mod tidy if you haven’t already done so.

The code sets up the necessary components for the Golang application. It loads the BPF program, attaches it to the specified network interface, and initializes the perf event reader. However, the code to read and process the perf events is yet to be implemented.

That concludes the content for the “Writing the Golang Application” section. Feel free to modify and customize the content according to your needs.

Building and Running the Project

Now that we have implemented the XDP eBPF program in C and the Golang application, let’s build and run the project.

Building the XDP eBPF Program

You can skip this step if you have already compiled the dilih_kern.o from the steps above.

Before building the XDP eBPF program, ensure that you have the necessary build tools and dependencies installed on your system. You can refer to the project’s README or documentation for the specific requirements.

To build the XDP eBPF program, navigate to the bpf directory and run the following command:

make

This command will compile the C code and generate the dilih_kern.o object file.

Building the Golang Application

To build the Golang application, make sure you are in the root directory of the project. Run the following command:

CGO_ENABLED=0 go build

This command will compile the Golang code and generate an executable binary file. Note: we do not need CGO for our application. We could leave it enabled if we wish, but I like to use CGO_ENABLED=0 when I can as it results in a statically compiled binary that I can easily load into containers.

Running the Go Project

sudo ./dilih

You should start to see a summary of the packets processed on the given interface:

Make sure to run the application with elevated privileges (sudo) to access the necessary resources.

The Golang application will start collecting data from the XDP program and display statistics based on the received perf events.

Cleaning Up To clean up the project and remove the XDP program from the network interface, run the following command:

sudo make clean

This command will detach the XDP program from the network interface and remove any associated artifacts.

That’s it! You have successfully built and run the project. Experiment with different network interfaces and observe the packet drop statistics displayed by the Golang application.

Feel free to explore additional features and modifications to enhance the project further.

Testing and Verifying the XDP eBPF Program

Testing and verifying the functionality of the XDP eBPF program is an essential step to ensure its correctness and effectiveness. In this section, we’ll cover some testing techniques and verification methods for the XDP program.

Test Environment Setup

To create a suitable test environment, we’ll utilize virtual network interfaces (veth devices) to simulate network traffic and observe the behavior of the XDP program.

Install the iproute2 package if it’s not already installed on your system. This package provides the necessary tools to manage network interfaces.

Create a pair of veth devices using the following commands:

sudo ip link add veth0 type veth peer name veth1

This command will create two virtual network interfaces (veth0 and veth1) that are connected to each other.

Set the interfaces up and assign IP addresses to them:

sudo ip link set veth0 up
sudo ip link set veth1 up
sudo ip addr add 10.0.0.1/24 dev veth0
sudo ip addr add 10.0.0.2/24 dev veth1

This will bring up the interfaces and assign IP addresses (10.0.0.1 and 10.0.0.2) to them.

With the veth devices set up, we can proceed to test and verify the XDP eBPF program’s functionality.

Packet Drop Verification

One of the primary functionalities of the XDP program is to drop a certain percentage of packets. We can verify this behavior by sending packets between the veth devices and observing the packet drop rate.

Open two terminal windows and navigate to the project directory in both of them.

In the first terminal, run the following command to listen for ICMP echo requests (ping):

sudo tcpdump -i veth1 icmp

In the second terminal, send ICMP echo requests (ping) from veth0 to veth1 using the following command:

sudo ip netns exec veth0 ping 10.0.0.2

Observe the output in the first terminal. You should see the ICMP echo requests being captured.

Analyze the packet capture to verify the packet drop rate. If the XDP program is working correctly, approximately 50% of the ICMP echo requests should be dropped, resulting in a reduced number of captured packets.

By performing packet drop verification tests, you can ensure that the XDP program is functioning as expected and dropping packets according to the specified percentage.

Performance Analysis

In addition to functional verification, it’s crucial to analyze the performance impact of the XDP eBPF program. This analysis helps evaluate the efficiency and overhead introduced by the program.

Use the provided Golang application to collect performance metrics and statistics from the XDP program. Refer to the “Building and Running the Project” section for instructions on how to run the Golang application.
Monitor and observe the average processing time for both passed and dropped packets. The Golang application displays the average processing time in nanoseconds (ns) for each packet type.

If the average processing time is consistently low, it indicates that the XDP program is performing efficiently and causing minimal processing overhead.

If the average processing time is significantly high, it may indicate that the XDP program is introducing a considerable processing overhead, which may require optimization or further investigation.
Collect data and analyze the performance metrics over an extended period of network traffic to identify any patterns or trends. Look for anomalies or deviations in the processing time that could indicate potential bottlenecks or inefficiencies.
Experiment with different packet drop percentages and observe their impact on the average processing time. By varying the drop rate, you can assess the trade-off between packet loss and processing efficiency.
Performing performance analysis allows you to gain insights into the impact of the XDP eBPF program on network performance and make informed decisions about its optimization and tuning.

Integration and System Testing To ensure the proper integration of the XDP eBPF program into the overall system, it’s essential to perform integration and system testing. This involves testing the interaction between the XDP program, the network stack, and other components of the system.

Construct a test scenario that closely resembles the production environment in which the XDP program will operate. Consider factors such as network traffic patterns, system load, and the presence of other networking components.

Generate realistic network traffic using tools such as packet generators, traffic simulators, or actual production traffic if available.

Monitor the system’s behavior, including packet processing, performance metrics, and system resource utilization. Ensure that the XDP program functions as expected and does not introduce any adverse effects on the system.

Test corner cases and edge conditions to validate the robustness and resilience of the XDP program. This includes scenarios such as high network traffic volumes, unusual packet structures, or unexpected network events.

By conducting integration and system testing, you can ensure that the XDP eBPF program seamlessly integrates into the broader system and operates reliably under various conditions.

Conclusion

In this article, we explored the process of building an XDP eBPF program with C and Golang. We started by understanding the basics of XDP and eBPF, followed by setting up the development environment and writing the XDP eBPF program in C. We then integrated the program with a Golang application to collect and analyze performance metrics.

Throughout the journey, we learned how to compile and load the XDP program, build the Golang application, and leverage the power of eBPF and XDP to manipulate network packets and introduce controlled chaos. We also discussed testing methodologies to ensure the correctness, efficiency, and integration of the XDP program within the system.

The ability to leverage eBPF and XDP opens up a world of possibilities for network programmability, performance optimization, and security enhancements. By harnessing the flexibility and programmability of eBPF, developers can create powerful and efficient networking applications.

We encourage you to explore further possibilities with XDP and eBPF, experiment with different scenarios, and dive deeper into the rich ecosystem of eBPF tools and libraries available. Embrace the power of XDP and eBPF to unlock new horizons in network programming and performance optimization.

Happy coding!

Additional Resources

To further expand your knowledge and explore the world of XDP, eBPF, and network programming, here are some valuable resources:

The Cilium Project: Cilium is an open-source project that provides networking and security capabilities powered by eBPF. Their documentation and codebase offer in-depth insights into eBPF and its applications. Visit their website at cilium.io for more information.
The iovisor Project: iovisor is an open-source project that focuses on building tools, libraries, and infrastructure for eBPF-based tracing, monitoring, and networking. Their website at iovisor.org hosts a wealth of resources, including tutorials, documentation, and sample code.
The BCC (BPF Compiler Collection): BCC is a collection of powerful command-line tools and libraries that leverage eBPF for various tracing and performance analysis tasks. The official GitHub repository at github.com/iovisor/bcc provides extensive documentation and examples to help you dive deep into eBPF.
eBPF.io: eBPF.io is a community-driven website dedicated to providing resources, tutorials, and news about eBPF. It features articles, case studies, and a curated list of tools and libraries related to eBPF. Explore the website at ebpf.io to stay up-to-date with the latest developments in the eBPF ecosystem.
Linux Kernel Documentation: The Linux kernel documentation includes a comprehensive section on eBPF and XDP, covering various aspects, including API references, usage examples, and implementation details. Access the documentation at www.kernel.org/doc/html/latest/bpf to gain a deep understanding of the underlying mechanisms.

These resources serve as valuable references and provide opportunities for further learning and exploration. Delve into the world of XDP, eBPF, and network programming, and unlock the full potential of these technologies in your networking projects.

You can also find the entire codebase for this article here: https://github.com/peter-mcconnell/dilih