Linux on Peter McConnell :: Ponderings from a Linux Systems engineer

Snooping on libpam (openssh auth, passwd) with Golang and eBPF

Sun, 18 Feb 2024 11:40:02 +0000

In the vast and complex landscape of software security, safeguarding sensitive information remains a paramount concern for developers and security professionals alike. Among the myriad of challenges, securely managing and protecting credentials during authentication processes stands out as a critical vulnerability point. Traditional security measures often fall short in providing real-time insights into how credentials are handled and potentially exposed within applications, especially those relying on widely used authentication frameworks like PAM (Pluggable Authentication Modules).

I work for a runtime security platform so this topic, BPF, Golang and Linux are some that I find the most interesting. Tangentally I was asked to give a talk at a Golang meetup, so figured I’d create a mini project to showcase some of this tech may be a good way to get others excited about these topics. Hence this repo and article.

The source code for this project can be seen here: https://github.com/peter-mcconnell/whispers.

A high-level introduction to some of the tech we’ll be playing with

Two pivotal technologies at the heart of our work here are uprobes (user space probes) and eBPF (Extended Berkeley Packet Filter).

Uprobes: User Space Probes

Uprobes are a dynamic tracing feature of the Linux kernel that allows developers to instrument user space binaries. By attaching to specific locations in the executable code (such as function entry or exit points), uprobes enable the collection of data about the binary’s execution without modifying its code. This makes uprobes an invaluable tool for performance analysis, debugging, and, as with whispers, security monitoring.

When a specified location is executed, the uprobe triggers, and control is passed to a handler function, which can gather information about the execution context, such as function arguments, return values, and the process ID. This capability allows whispers to monitor authentication processes by attaching uprobes to critical functions within libpam, capturing credential data as it’s passed through these functions.

eBPF: Extended Berkeley Packet Filter

eBPF is a revolutionary technology that extends the traditional Berkeley Packet Filter (BPF) with enhanced capabilities, allowing for the safe execution of small programs within the Linux kernel without changing kernel source code or loading kernel modules. eBPF programs are written in a restricted C-like language, compiled into bytecode, and then executed by the kernel’s virtual machine, ensuring safety and efficiency.

eBPF has a wide range of applications, from networking and security to performance monitoring and debugging. It interacts with the kernel and user space through maps (data structures for storing state) and program types (defining what an eBPF program can do).

For our project, eBPF is used to implement uprobes for capturing authentication data. The eBPF program is attached to the target function in the user space binary, and upon function execution, the eBPF program is triggered, capturing relevant data and passing it back to user space for analysis. This seamless interaction between user and kernel space is what enables whispers to monitor and log credential information efficiently and transparently. The combination of uprobes and eBPF offers a powerful mechanism for introspecting and monitoring system and application behavior in real-time. With uprobes providing the ability to hook into specific points of interest in user space applications and eBPF facilitating the safe execution of custom logic within the kernel, developers can create sophisticated monitoring tools like whispers that are both efficient and minimally invasive.

This innovative approach to system monitoring not only enhances security by exposing potential vulnerabilities and data exposures but also serves as an educational tool, shedding light on the inner workings of system authentication processes and the capabilities of modern Linux kernel technologies.

BPF Maps: Bridging Kernel and User Space

An integral component of eBPF’s powerful capabilities comes from BPF maps, sophisticated data structures designed to store and share data between eBPF programs running in the kernel and applications in user space. These maps play a crucial role in maintaining state, passing information, and facilitating complex data management tasks within eBPF-driven applications like whispers. What Are BPF Maps?

BPF maps are key-value stores that can be accessed by eBPF programs and user space applications. They come in various types, each optimized for specific use cases, such as arrays, hash maps, and ring buffers. The choice of map type depends on the nature of the data being stored and the access patterns required by the eBPF program and the user space application.

Key Features and Benefits:

Efficient Data Sharing: BPF maps provide a high-performance mechanism for sharing data between the kernel and user space, crucial for applications that require real-time processing and analysis.
Statefulness: Unlike traditional stateless packet processing in the kernel, BPF maps enable eBPF programs to maintain state across function calls and packet processing stages, allowing for more sophisticated logic and tracking capabilities.
Versatility: The variety of available map types makes BPF maps suitable for a wide range of applications, from performance monitoring and networking to security and observability tools like whispers.
Safety and Isolation: Access to BPF maps from eBPF programs is rigorously checked by the kernel’s verifier, ensuring that only valid, safe operations are performed, thereby protecting the kernel from potentially malicious or buggy code.

In the context of our project “whispers”, BPF maps serve as the primary mechanism for transferring captured credential data from the eBPF program (attached to libpam functions via uprobes) to the user space component of the tool. For instance, ring buffers are utilized to efficiently pass event data (such as authentication attempts) from kernel to user space, where it can be processed, analyzed, and logged. This enables whispers to monitor and report on authentication processes with minimal overhead and without requiring direct access to the monitored application’s memory or internal data structures.

Knowing what to look for

We know that we want to snoop for creds on libpam, as used by OpenSSH. In order to “snoop” on libpam we need to know which functions and structures of this library we should snoop on. We also know that we wish to attach uprobes, so are relying on the available symbols of libpam - that seems a reasonable place to start.

Creating a debug environment

When setting off on a new adventure I like to create a little investigation environment that allows me to poke at things, break them etc. Often this results in a Dockerfile and this case is no different:

FROM ubuntu:22.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update -yq && \
		apt-get install -yq openssh-server && \
		# some debug utilities, to aide exploration
		apt-get install -yq binutils bpftrace systemtap systemtap-sdt-dev linux-headers-$(uname -r) vim && \
		mkdir -p /var/run/sshd && \
		echo 'root:pass' | chpasswd && \
		sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \
		echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

This not only installs and configures openssh, but also includes some toys for debugging. Using this Dockerfile I docker build -t=debug . and docker run --privileged -p 2222:22 --rm --name debug -d debug to get it going in the background. I then docker exec -ti debug bash to poke around. We’re running --privileged so that we can test attaching probes and running -p 2222:22 so we can attempt to ssh into the container from host.

The screenshot above shows the simple process of hunting down where libpam is using sshd as a starting point. I could have just done find / -name "libpam*" but I wanted to be extra sure it was linked from openssh, which is what I’m intending on using for my tests.

Now that I’ve identified the location for libpam, and that it is indeed linked from sshd, I can proceed to identify symbols which may be of interest. To do this I simply ask readelf to dump the symbols and grep the output for auth which is a term I’d expect to see in the symbol name for a function that handles authentication.

root@664e76f33de2:/# readelf -Ws /lib/x86_64-linux-gnu/libpam.so.0 | grep auth
    82: 00000000000088e0   699 FUNC    GLOBAL DEFAULT   15 pam_get_authtok_verify@@LIBPAM_EXTENSION_1.1.1
    93: 00000000000088b0    12 FUNC    GLOBAL DEFAULT   15 pam_get_authtok@@LIBPAM_EXTENSION_1.1
   112: 00000000000088c0    26 FUNC    GLOBAL DEFAULT   15 pam_get_authtok_noverify@@LIBPAM_EXTENSION_1.1.1
   116: 0000000000009ab0   371 FUNC    GLOBAL DEFAULT   15 pam_chauthtok@@LIBPAM_1.0
   118: 0000000000009940   259 FUNC    GLOBAL DEFAULT   15 pam_authenticate@@LIBPAM_1.0

These all seem reasonably interesting. To ensure these are the right things to hook onto I test them out using `bpftrace -e ‘uprobe:/lib/x86_64-linux-gnu/libpam.so.0:pam_get_authtok { printf(“pam_get_authtok called\n”); }’

With bpftrace running and the uprobe trace attached I then ssh into the container using ssh root@localhost -p 2222:22 from another terminal and can see the trace is made when authentication is handled. Seems like we’re on the right track.

Now I have something to look for in the source code so head over to https://github.com/linux-pam/linux-pam/ where after a bit of searching for the symbols I find https://github.com/linux-pam/linux-pam/blob/1e2c6cecf81dcaeea0c2c9d37bc35eea120cd77d/libpam/pam_get_authtok.c#L213 which looks like the function signature we’re after:

int
pam_get_authtok (pam_handle_t *pamh, int item, const char **authtok,
		 const char *prompt)
{
  return pam_get_authtok_internal (pamh, item, authtok, prompt, 0);
}

With our BPF program we’ll intercept the arguments, so first thing to look up is pam_handle_t which after some scouring leads to https://github.com/linux-pam/linux-pam/blob/1e2c6cecf81dcaeea0c2c9d37bc35eea120cd77d/libpam/pam_private.h#L154:


struct pam_handle {
    char *authtok;
    unsigned caller_is;
    struct pam_conv *pam_conversation;
    char *oldauthtok;
    char *prompt;                /* for use by pam_get_user() */
    char *service_name;
    char *user;
    char *rhost;
    char *ruser;
    char *tty;
    char *xdisplay;
    char *authtok_type;          /* PAM_AUTHTOK_TYPE */
    struct pam_data *data;
    struct pam_environ *env;      /* structure to maintain environment list */
    struct _pam_fail_delay fail_delay;   /* helper function for easy delays */
    struct pam_xauth_data xauth;        /* auth info for X display */
    struct service handlers;
    struct _pam_former_state former;  /* library state - support for
					 event driven applications */
    const char *mod_name;	/* Name of the module currently executed */
    int mod_argc;               /* Number of module arguments */
    char **mod_argv;            /* module arguments */
    int choice;			/* Which function we call from the module */

#ifdef HAVE_LIBAUDIT
    int audit_state;             /* keep track of reported audit messages */
#endif
    int authtok_verified;
    char *confdir;
};

This has some interesting fields like authtok and user. At this point it feels like we have a reasonable direction for what to hook into.

Developing the eBPF Program for whispers

This section provides a high level overview at the development of the eBPF program for whispers, a tool designed to monitor and capture credentials handled by libpam. By leveraging eBPF, whispers taps into kernel functions related to authentication processes, offering unprecedented insights into credential management and security vulnerabilities. You can check out all of the source code here: https://github.com/peter-mcconnell/whispers.

Dependencies

Before diving into eBPF program development, ensure your development environment is properly set up with the following tools and libraries:

LLVM & Clang: For compiling eBPF programs.
Linux Kernel Headers: Required for eBPF program compilation to access kernel APIs.
libbpf: A library for loading and interacting with eBPF programs and maps.
Go toolchain: For the Go-based components of whispers.

Developing the eBPF Program

The eBPF program for whispers is designed to attach to the pam_get_authtok function via a uretprobe, capturing credentials as they’re processed.

uretprobe.c: Contains the eBPF code that defines the data structures and logic for capturing and processing credential information from pam\_get\_authtok.

When trying to look up the structure of pam_get_authtok I encountered https://github.com/citronneur/pamspy which is where I lifted most of this code from, so credit to them, but you can see this just matches the structure we found in the libpam source code mentioned above:

https://github.com/peter-mcconnell/whispers/blob/main/bpf/uretprobe.c

...

// we need the pam_handle struct so that we can read the data in our BPF program
typedef struct pam_handle
{
  char *authtok;
  unsigned caller_is;
  void *pam_conversation;
  char *oldauthtok;
  char *prompt;
  char *service_name;
  char *user;
  char *rhost;
  char *ruser;
  char *tty;
  char *xdisplay;
  char *authtok_type;
  void *data;
  void *env;
} pam_handle_t;

// we're using a ringbuffer datastructure to share data back to our userspace program
struct
{
  __uint(type, BPF_MAP_TYPE_RINGBUF);
  __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

// and we will use this struct to store the items for our bpf map
typedef struct _event_t {
    int  pid;
    char comm[16];
    char username[80];
    char password[80];
} event_t;
...

SEC("uretprobe/pam_get_authtok")
int trace_pam_get_authtok(struct pt_regs *ctx)
{
  if (!PT_REGS_PARM1(ctx))
    return 0;

  pam_handle_t* phandle = (pam_handle_t*)PT_REGS_PARM1(ctx);

  u32 pid = bpf_get_current_pid_tgid() >> 32;

  u64 password_addr = 0;
  bpf_probe_read(&password_addr, sizeof(password_addr), &phandle->authtok);

  u64 username_addr = 0;
  bpf_probe_read(&username_addr, sizeof(username_addr), &phandle->user);

  event_t *e;
  e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
  if (e)
  {
    e->pid = pid;
    bpf_probe_read(&e->password, sizeof(e->password), (void *)password_addr);
    bpf_probe_read(&e->username, sizeof(e->username), (void *)username_addr);
    bpf_get_current_comm(&e->comm, sizeof(e->comm));
    bpf_ringbuf_submit(e, 0);
  }
  return 0;
};

To handle compiling this code, given we’re going to be using Golang we’ll define a go generate later which will point to a bash script which will handle compilation. For now though we can create the gen.sh which will handle the compilation:

#!/usr/bin/env bash

set -e
set -x

# Dynamically find kernel headers. Adjust this line according to your needs.
KERNEL_HEADERS=$(find /usr/src -name "linux-headers-$(uname -r)" -type d | head -n 1)

# Include directory for bpf_helpers.h. This might need adjustments.
BPF_HELPERS_DIR="${KERNEL_HEADERS}/tools/bpf/resolve_btfids/libbpf/include/"

# Run bpf2go with dynamic include paths
go run github.com/cilium/ebpf/cmd/bpf2go -target amd64 bpf \
    ../../bpf/uretprobe.c -- \
    -I"${KERNEL_HEADERS}" \
    -I"${BPF_HELPERS_DIR}" \
    -I../../bpf/headers

This will be invoked by go build whenever it encounters a // go generate ../path/to/gen.sh directive. go generate is a command provided by the Go toolchain that automates the generation of code before the build process. It scans Go source files for special comments that specify commands to be run at generate time. These commands can invoke any process but are commonly used to generate Go code, making it a perfect fit for integrating eBPF programs into Go applications.

Golang code

This section delves into the Go component of whispers, which serves as the user-space counterpart to the eBPF program. It outlines how to leverage Go to load the eBPF program, attach probes, read from eBPF maps, and process the captured credential data for monitoring and analysis purposes.

Ensure your Go environment is set up with the necessary dependencies:

Go (1.21 or newer): Ensure the Go toolchain is installed.

The Go codebase for whispers is structured as follows:

cmd/: Contains the CLI interface for whispers.
pkg/config: Defines configuration structures.
pkg/whispers: Implements the core functionality for loading and interacting with the eBPF program.

Dependencies are defined in go.mod and can be pulled with go mod tidy.

This article will only touch on the most interesting parts of the code. The source can be read at https://github.com/peter-mcconnell/whispers/blob/main/pkg/whispers/whispers.go.

We’re using cilium ebpf to handle loading our BPF program and interacting with the BPF maps. In the following, bpfObjects and loadBpfObjects are defined in the auto-generated file bpf_x86_bpfel.go which is generated during go build due to the // go:generate ... directive which triggers gen.sh:

func Listen(ctx context.Context, cfg *config.Config) error {
    objs := bpfObjects{}
    if err := loadBpfObjects(&objs, nil); err != nil {
        return err
    }
    defer objs.Close()

    ...
}

The above snippet demonstrates loading the eBPF program and ensuring it is correctly cleaned up upon termination.

whispers attaches uretprobes to target functions (e.g., pam_get_authtok) also using the cilium/ebpf library:

ex, err := link.OpenExecutable(cfg.BinPath)
if err != nil {
    return err
}
up, err := ex.Uretprobe(cfg.Symbol, objs.TracePamGetAuthtok, nil)
if err != nil {
    return err
}
defer up.Close()
...

The Go code reads from the BPF map (a ring buffer) to read events captured by the eBPF program:

rb, err := ringbuf.NewReader(objs.Rb)
if err != nil {
    log.Printf("failed to open ring buffer reader: %v", err)
}
defer rb.Close()

go func() {
    for {
        record, err := rb.Read()
        ...
        event := parseEventData(record.RawSample)
        ...
    }
}()

We’ll need to handle reading the raw data from the BPF map and converting it into something we can easily interact with in Golang:

...
// this needs to perfectly match the structure of our BPF map items (defined in our .h file as _event_t)
type eventT struct {
	Pid      int32
	Comm     [16]byte
	Username [80]byte
	Password [80]byte
}
...
func byteArrayToString(b []byte) string {
	n := -1
	for i, v := range b {
		if v == 0 {
			n = i
			break
		}
	}
	if n == -1 {
		n = len(b)
	}
	return string(b[:n])
}

func parseEventData(data []byte) *eventT {
	var event eventT

	if len(data) >= int(unsafe.Sizeof(eventT{})) {
		if err := binary.Read(bytes.NewReader(data), binary.LittleEndian, &event); err == nil {
			return &event
		}
	}

	return nil
}
...

After reading events from the ring buffer, whispers processes and logs the captured credentials:

log.Printf("Event: PID: %d, Comm: %s, Username: %s, Password: %s",
    event.Pid,
    byteArrayToString(event.Comm[:]),
    byteArrayToString(event.Username[:]),
    byteArrayToString(event.Password[:]))

Seeing it in action

Included with the repository is a Dockerfile that not only builds whisper but also contains an sshd server ready to go, so that you can play with it immediately. The following is a demonstration of that:

Improvements

This project was designed to be very simple - specifically I’m aiming to give an overview of it during a tech talk. As such, it isn’t intended to be feature-complete. Namely this project results in a binary whereby you provide the -binPath=/path/to/libpam, however we can automate this attachment with more eBPF.

This project has used uprobes to trace arguments for a given user-space application. However we can automate the auto-attachment of these probes using kprobes. With kprobes we could filter execve and fork for comms that we deem to be interesting (e.g. sshd, passwd) and when we get a match, we could run ldd against the /proc/<pid>/exe to see if we can see libpam and attach that way. There are some complexities to consider here such as the namespace for a given process may be different (e.g. for containers). We’d likely also want to look at existing processes and attach probes to those also. However, these are all trivial problems to solve. The result being that you could run a single whispers with no targets, that would automatically attach our uprobes whenever a suitable process was started.

Docker Overlayfs: How filesystems work in Docker

Thu, 19 Jan 2023 12:05:02 +0000

This is a brief follow up to my article on Docker networking: Network Namespaces, Docker Bridge and DNS

Docker uses OverlayFS to manage the file system of its containers. When a container is run, Docker creates a new layer for the container’s file system on top of the base image. This allows the container to have its own file system that is isolated from the host system and other containers.

Running the ubuntu:22.04 image we can see the root file system differs from the host where I’m running it. Below you can see there is a file in root called /.dockerenv:

$ docker run --rm -ti ubuntu:22.04 bash
root@541cc3b62543:/# ls -al /
total 56
drwxr-xr-x   1 root root 4096 Jan 19 11:51 .
drwxr-xr-x   1 root root 4096 Jan 19 11:51 ..
-rwxr-xr-x   1 root root    0 Jan 19 11:51 .dockerenv
lrwxrwxrwx   1 root root    7 Nov 30 02:04 bin -> usr/bin
drwxr-xr-x   2 root root 4096 Apr 18  2022 boot
drwxr-xr-x   5 root root  360 Jan 19 11:51 dev
drwxr-xr-x   1 root root 4096 Jan 19 11:51 etc
drwxr-xr-x   2 root root 4096 Apr 18  2022 home
lrwxrwxrwx   1 root root    7 Nov 30 02:04 lib -> usr/lib
lrwxrwxrwx   1 root root    9 Nov 30 02:04 lib32 -> usr/lib32
lrwxrwxrwx   1 root root    9 Nov 30 02:04 lib64 -> usr/lib64
lrwxrwxrwx   1 root root   10 Nov 30 02:04 libx32 -> usr/libx32
drwxr-xr-x   2 root root 4096 Nov 30 02:04 media
drwxr-xr-x   2 root root 4096 Nov 30 02:04 mnt
drwxr-xr-x   2 root root 4096 Nov 30 02:04 opt
dr-xr-xr-x 491 root root    0 Jan 19 11:51 proc
drwx------   2 root root 4096 Nov 30 02:07 root
drwxr-xr-x   5 root root 4096 Nov 30 02:07 run
lrwxrwxrwx   1 root root    8 Nov 30 02:04 sbin -> usr/sbin
drwxr-xr-x   2 root root 4096 Nov 30 02:04 srv
dr-xr-xr-x  13 root root    0 Jan 19 11:51 sys
drwxrwxrwt   2 root root 4096 Nov 30 02:07 tmp
drwxr-xr-x  14 root root 4096 Nov 30 02:04 usr
drwxr-xr-x  11 root root 4096 Nov 30 02:07 var

Which does not exist at root on the host running the container:

root@541cc3b62543:/# 
root@541cc3b62543:/# exit
exit
$ stat /.dockerenv
stat: cannot statx '/.dockerenv': No such file or directory

So … where does it exist? To inspect the layers of a running container, you can use the “docker inspect” command followed by the container ID or name. This will return a JSON object containing information about the container, including its layers. To view this we’ll re-run our ubuntu:22.04 container, grab the ID and inspect it:

$ docker run --rm -d -ti ubuntu:22.04 bash
6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb
# lower directory
$ docker inspect 6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb -f '{{.GraphDriver.Data.LowerDir}}'
/dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a-init/diff:/dockerstore/overlay2/bb9057b4f1980fe004301f181c3313c15c2a75b7c7b7c5a6fe80159d2275f0d3/diff

# upper directory
$ docker inspect 6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb -f '{{.GraphDriver.Data.UpperDir}}'
/dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/diff

# merged directory
$ docker inspect 6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb -f '{{.GraphDriver.Data.MergedDir}}'
/dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/merged

I’ll keep this container running and we’ll dig into these contents shortly.

When a container is run, its layers are stored in the host system’s file system, typically in the /var/lib/docker/overlay2 directory. You can see mine is in /dockerstore/ as I have manually set data-root in /etc/docker/daemon.json for the host that I’m testing this on. Each layer is represented by a directory that contains the files and directories that make up that layer. The topmost layer is the one that the container is currently using, and the lower layers are the ones that are inherited from the base image.

The advantages of using layers in Docker include:

Smaller image size, since multiple containers can share a common base image
Faster container startup time, since only the changes made to the container are stored in new layers
Easier to manage and update containers, since changes can be made to a container’s layer without affecting the base image
Greater security, since each container’s file system is isolated from other containers and the host system.

Please keep in mind that the information is general and may vary depending on specific scenarios.

Now lets take a deeper look at the filesystem for our running container.

LowerDir

This value is unique in the outputs above in that it’s actually two paths, separated by a colon:

$ docker inspect 6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb -f '{{.GraphDriver.Data.LowerDir}}'
/dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a-init/diff:/dockerstore/overlay2/bb9057b4f1980fe004301f181c3313c15c2a75b7c7b7c5a6fe80159d2275f0d3/diff

The first part (left side of :) is the path to the init layer of the container. this is the layer that contains the initial filesystem of the container, which is based on the base image. We can take a look at the contents of that layer with ls:

sudo ls /dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a-init/diff
dev  etc

The second part (right side of :) is the path to the layer of the container that includes changes from the rest of the Dockerfile. Again we can take a look:

sudo ls /dockerstore/overlay2/bb9057b4f1980fe004301f181c3313c15c2a75b7c7b7c5a6fe80159d2275f0d3/diff
bin  boot  dev	etc  home  lib	lib32  lib64  libx32  media  mnt  opt  proc  root  run	sbin  srv  sys	tmp  usr  var

To better visualise this, lets create our own Dockerfile:

FROM ubuntu:22.04

RUN mkdir -p /testinglowerdir/ && echo -n "hellothere" > /testinglowerdir/foo

Now, given what we learned above, when we run this container the first part of LowerDir should contain all the contents for ubuntu:22.04 and the second part of LowerDir should contain only /testinglowerdir/:

$ docker build -t=test .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ubuntu:22.04
 ---> 6b7dfa7e8fdb
Step 2/2 : RUN mkdir -p /testinglowerdir/ && echo -n "hellothere" > /testinglowerdir/foo
 ---> Running in e71a7cd5541c
Removing intermediate container e71a7cd5541c
 ---> df924945a2b0
Successfully built df924945a2b0
Successfully tagged test:latest
$ docker run --rm -d -ti test bash
9c9fe0bcd283bc0c9649b77246115e3a09e8885efd53f0e9de09de537bea9188
$ docker inspect 9c9fe0bcd283bc0c9649b77246115e3a09e8885efd53f0e9de09de537bea9188 -f '{{.GraphDriver.Data.LowerDir}}'
/dockerstore/overlay2/5501fd185b14a60317f3e0db485bb8f8c5cf41b7cb1ed0688526ba918938b7bf-init/diff:/dockerstore/overlay2/4d49e9a62bad55c3761ab08ded87f56010b28a40f264896c01e5c1c653b826a8/diff:/dockerstore/overlay2/bb9057b4f1980fe004301f181c3313c15c2a75b7c7b7c5a6fe80159d2275f0d3/diff
$ # show directory contents for second part of LowerDir
$ sudo ls /dockerstore/overlay2/4d49e9a62bad55c3761ab08ded87f56010b28a40f264896c01e5c1c653b826a8/diff
testinglowerdir

UpperDir

$ docker inspect 6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb -f '{{.GraphDriver.Data.UpperDir}}'
/dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/diff

The UpperDir contains changes that we’ve made at runtime. To see this in action we can exec into our container and create a simple directory with a file in the root directory:

docker exec -ti 6a9 bash
root@6a9014d7ebfd:/# mkdir /tutorial
root@6a9014d7ebfd:/# echo 'iseeyou' > /tutorial/ohai

We can now see this in our UpperDir directory:

$ sudo ls /dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/diff/
root  tutorial
$ sudo cat /dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/diff/tutorial/ohai
iseeyou

Want to quickly see what files are being created by a running container? This is something the UpperDir can tell you.

MergedDir

$ docker inspect 6a9014d7ebfddb3a107b29aca3764f24e51f64fda1e8b8cec135c18923daefeb -f '{{.GraphDriver.Data.MergedDir}}'
/dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/merged

I’m sure you’ve guessed what this one is… This is the merged structure:

$ sudo ls /dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/merged
bin  boot  dev	etc  home  lib	lib32  lib64  libx32  media  mnt  opt  proc  root  run	sbin  srv  sys	tmp  tutorial  usr  var

Here you can see all of the directories from the LowerDir and UpperDir together. We can chroot into this directory to “see what docker sees”:

sudo chroot /dockerstore/overlay2/268eb11c54948d6293aa3947b7a2c83b1395b18509518e26487f0e79997f787a/merged /bin/bash
root@pete:/# ls -al
total 72
drwxr-xr-x  1 root root 4096 Jan 19 12:21 .
drwxr-xr-x  1 root root 4096 Jan 19 12:21 ..
-rwxr-xr-x  1 root root    0 Jan 19 11:56 .dockerenv
lrwxrwxrwx  1 root root    7 Nov 30 02:04 bin -> usr/bin
drwxr-xr-x  2 root root 4096 Apr 18  2022 boot
drwxr-xr-x  1 root root 4096 Jan 19 11:56 dev
drwxr-xr-x  1 root root 4096 Jan 19 11:56 etc
drwxr-xr-x  2 root root 4096 Apr 18  2022 home
lrwxrwxrwx  1 root root    7 Nov 30 02:04 lib -> usr/lib
lrwxrwxrwx  1 root root    9 Nov 30 02:04 lib32 -> usr/lib32
lrwxrwxrwx  1 root root    9 Nov 30 02:04 lib64 -> usr/lib64
lrwxrwxrwx  1 root root   10 Nov 30 02:04 libx32 -> usr/libx32
drwxr-xr-x  2 root root 4096 Nov 30 02:04 media
drwxr-xr-x  2 root root 4096 Nov 30 02:04 mnt
drwxr-xr-x  2 root root 4096 Nov 30 02:04 opt
drwxr-xr-x  2 root root 4096 Apr 18  2022 proc
drwx------  1 root root 4096 Jan 19 12:16 root
drwxr-xr-x  5 root root 4096 Nov 30 02:07 run
lrwxrwxrwx  1 root root    8 Nov 30 02:04 sbin -> usr/sbin
drwxr-xr-x  2 root root 4096 Nov 30 02:04 srv
drwxr-xr-x  2 root root 4096 Apr 18  2022 sys
drwxrwxrwt  2 root root 4096 Nov 30 02:07 tmp
drwxr-xr-x  2 root root 4096 Jan 19 12:20 tutorial
drwxr-xr-x 14 root root 4096 Nov 30 02:04 usr
drwxr-xr-x 11 root root 4096 Nov 30 02:07 var
root@pete:/# cat /tutorial/ohai 
iseeyou
root@pete:/#

Pretty sweet! Another way / a “better” way that we can get this view is with nsenter:

$ sudo nsenter --target $(docker inspect --format {{.State.Pid}} 6a9) --mount --uts --ipc --net --pid
root@6a9014d7ebfd:/# cat /tutorial/ohai 
iseeyou
root@6a9014d7ebfd:/#

Do it yourself

This has been a quick look into how Docker avails of OverlayFS, but you can of course do this yourself. The basic syntax is:

mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=workdir target

lowerdir is the lower filesystem
upperdir is the upper filesystem
workdir is a directory where the OverlayFS stores metadata about the overlay
target is the mount point where the overlay will be mounted

For example, if you have two directories, /mnt/lower and /mnt/upper, you can create an OverlayFS file system that combines them at /mnt/overlay with the following command:

mount -t overlay overlay -o lowerdir=/mnt/lower,upperdir=/mnt/upper,workdir=/mnt/workdir /mnt/overlay

To view the contents of the overlay, you can simply navigate to the mount point (in this example, /mnt/overlay) and use standard Linux commands to view the files and directories.

You can also use lsblk command to view the mounted overlays in your system and also you can unmount the overlays using umount command.

Please keep in mind that this is a basic example and there are many other options and settings that can be used when creating an OverlayFS file system.

Docker networking: Network Namespaces, Docker Bridge and DNS

Wed, 18 Jan 2023 19:48:46 +0000

Ever wondered how docker compose lets you communicate between services? This article takes a high level look at network namespaces, Dockers internal DNS and Docker bridge.

Network namespaces are a powerful feature in Linux that allows for the isolation of network stacks, creating multiple virtual networks on a single host. This concept is particularly useful for scenarios such as containerization, where each container needs its own independent network stack. In this article we’ll take a look at how docker / docker compose utilize this technology to grant containers network isolation and also take a look at how docker handles cross-container networking.

simple docker run

To see network namespaces in action, we’ll run a simple container (without the --network=host flag):

# run nginx in detached mode on host port 8080
docker run --name dummynginx -p 8080:80 -d nginx

If we ran this with --network=host it would share the network namespace of the host and therefore would not have it’s own network namespace.

Now we can check to see if we any network namespaces were created for this container:

$ sudo lsns -t net | grep nginx
[sudo] password for pete: 
        NS TYPE NPROCS   PID USER     NETNSID NSFS                           COMMAND
...
4026533009 net      17 52981 root           0 /run/docker/netns/3161e4b47f76 nginx: master process nginx -g daemon off;

In the output above we can see a network namespace was created for a docker nsfs with the command nginx. This seems like a likely culprit. We could confirm this by checking for the nsfs in docker inspect <container id>.

Each container runs in its own network namespace, which means it has its own set of network interfaces, IP addresses, and routing tables. This isolation allows each container to have its own network configuration, without interfering with the host’s network or other containers. You can view these resources from the network namespace by using the ip command with nsenter.

In addition, the isolation also allows you to use the same port on the host for different services running in different containers. This is useful when you want to run multiple instances of the same service on a single host, each with its own IP address and port.

Also, this isolation helps in providing security boundaries between different containers, as they can’t see or interact with each other’s network stack unless explicitly configured to do so.

docker compose

Let’s remove that container and see how things look with docker compose which can allow multiple containers to communicate with eachother:

# remove the old nginx container
docker rm -f dummynginx

Now we’ll create a simple docker-compose.yaml:

---
version: '3.1'
services:
  nginx_a:
    image: 'nginx:latest'
    ports:
      - '8080:80'
  nginx_b:
    image: 'nginx:latest'

With this file stored as docker-compose.yaml we’ll spin it up then check lsns:

$ docker compose up -d
[+] Running 3/3
 ⠿  Network linuxnetworks_default      Created  0.0s
 ⠿ Container linuxnetworks-nginx_b-1   Started  0.3s
 ⠿ Container linuxnetworks-nginx_a-1   Started  0.6s
$ sudo lsns -t net | grep nginx
[sudo] password for pete: 
        NS TYPE NPROCS   PID USER     NETNSID NSFS                           COMMAND
...
4026533008 net      17 57327 root           0 /run/docker/netns/20790e4f73be nginx: master process nginx -g daemon off;
4026533108 net      17 57501 root           1 /run/docker/netns/11b21af68bd5 nginx: master process nginx -g daemon off;

Two network namespaces; one for each container. It’s worth noting if we had replicas in each service, each container within that service would get their own network namespace. So how does docker compose allow nginx_a to speak to nginx_b given each container has it’s own network namespace?

Firstly lets clarify the mystery we’re trying to solve:

docker compose exec nginx_a curl nginx_b
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

We connected to the nginx_a container, then curl’ed nginx_b. How does it work? We’ll dig in over the next few sections to understand how Docker makes this possible.

a quick note on the ’links:’ property

docker compose supports a links: property. This is not required to allow services to speak to each other. It’s main purpose is to allow you to create aliases or to use hostname resolution, linking between containers in different docker compose files.

more info: https://docs.docker.com/compose/compose-file/#links

dockers internal DNS server

Docker runs its own internal DNS server at 127.0.0.11. How do we know this? We can see it:

docker compose exec -ti nginx_a bash
root@b315617ce0dd:/# cat /etc/resolv.conf 
nameserver 127.0.0.11
options ndots:0
root@b315617ce0dd:/#

We can query this from nginx_a by installing dnsutils and using nslookup or dig:

docker compose exec -ti nginx_a bash
root@b315617ce0dd:/# apt update && apt install dnsutils
root@b315617ce0dd:/# nslookup nginx_b 127.0.0.11
Server:         127.0.0.11
Address:        127.0.0.11#53

Non-authoritative answer:
Name:   nginx_b
Address: 172.21.0.2

root@b315617ce0dd:/#

Just to be sure that IP is correct we can validate the IP for nginx_b with docker inspect:

docker inspect $(docker ps -f 'Name=nginx_b' -q) -f '{{.NetworkSettings.Networks.linuxnetworks_default.IPAddress}}'
172.21.0.2

Looks good.

When a container is created and connected to a network, Docker automatically creates a DNS record for that container, using the container name as the hostname and the IP address of the container as the record’s value. This enables other containers on the same network to access each other by name, rather than needing to know the IP address of the target container.

virtual network interfaces

When we ran docker compose it created a virtual network interface inside each containers network namespace. To look at the resources in a network namespace we first need to know where the nsfs is. Looking at the example output above from lsns above we seen this line:

4026533008 net      17 57327 root           0 /run/docker/netns/20790e4f73be nginx: master process nginx -g daemon off;

You’ll see in the nsfs column we have: /run/docker/netns/20790e4f73be. With this we can enter the namespace and run some commands from that namespace.

You can also get this from docker itself. Running docker ps you can get the CONTAINER ID:

docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS          PORTS                                   NAMES
b315617ce0dd   nginx:latest   "/docker-entrypoint.…"   16 minutes ago   Up 16 minutes   0.0.0.0:8080->80/tcp, :::8080->80/tcp   linuxnetworks-nginx_a-1
da9caf14c90c   nginx:latest   "/docker-entrypoint.…"   16 minutes ago   Up 16 minutes   80/tcp                                  linuxnetworks-nginx_b-1

Then you can run inspect to see the network settings:

docker inspect b315617ce0dd -f '{{.NetworkSettings.SandboxKey}}'
/var/run/docker/netns/11b21af68bd5

Or the terribly ugly:

docker inspect $(docker ps -q -f 'Name=nginx_a') -f '{{.NetworkSettings.SandboxKey}}'

Armed with the nsfs path we can enter the namespace and run commands. Lets take a quick look at what interfaces are available in this namespace:

sudo nsenter --net=/run/docker/netns/11b21af68bd5 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
47: eth0@if48: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:15:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0

Cool! We’re finally peaking behind the curtain.

exploring the namespace

Now that we have the ability to run commands from the namespace we can request some more information about this interface with ip addr show:

sudo nsenter --net=/run/docker/netns/11b21af68bd5 ip addr show eth0
47: eth0@if48: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:15:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.21.0.3/16 brd 172.21.255.255 scope global eth0
       valid_lft forever preferred_lft forever

We can also get the route table information with ip route show:

sudo nsenter --net=/run/docker/netns/11b21af68bd5 ip route show
default via 172.21.0.1 dev eth0 
172.21.0.0/16 dev eth0 proto kernel scope link src 172.21.0.3

And we can even run good old faithful netstat to check out which ports we’re listening on:

sudo nsenter --net=/run/docker/netns/11b21af68bd5 netstat -tulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.11:32889        0.0.0.0:*               LISTEN      16568/dockerd       
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      57501/nginx: master 
tcp6       0      0 :::80                   :::*                    LISTEN      57501/nginx: master 
udp        0      0 127.0.0.11:51543        0.0.0.0:*                           16568/dockerd

docker bridge

Using the ip route command above we seen:

sudo nsenter --net=/run/docker/netns/11b21af68bd5 ip route show
default via 172.21.0.1 dev eth0 
172.21.0.0/16 dev eth0 proto kernel scope link src 172.21.0.3

So what is 172.21.0.1? That is the default gateway, the docker bridge. How can we know? Well, firstly we can list docker networks with:

$ docker network ls
NETWORK ID     NAME                    DRIVER    SCOPE
a52a287eb9f2   bridge                  bridge    local
8f5afced6d3f   host                    host      local
19508f629e30   none                    null      local

And then we can inspect the networks, like so:

$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "a52a287eb9f2af463e14ac4a97583b96a9dc66e28b5ac67967321386834e4e24",
        "Created": "2023-01-18T20:00:03.09775174Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

We can see the Gateway for bridge is set to 172.17.0.1.

The Docker bridge default gateway is responsible for connecting containers running in separate network namespaces, using the IP addresses and MAC addresses of the containers connected to it to forward packets to the correct container. When a packet is received by the bridge, it checks the destination IP address of the packet and compares it to the IP addresses of the connected containers. If a match is found, the packet is then forwarded to the corresponding container using the container’s MAC address. Additionally, the bridge also uses network address translation (NAT) to rewrite the source IP address of the packet to that of the bridge, allowing the containers to communicate with external networks.

so … how does nginx_a speak to nginx_b? (recap)

The request starts from nginx_a with curl nginx_a. nginx_a is configured to resolve DNS with Dockers internal DNS, as defined in its /etc/resolv.conf file. Dockers Internal DNS resolves nginx_b to 172.21.0.2 which is within the range of the default gateway in, the docker bridge, as defined in this containers route table. The Docker Bridge acts as a gateway and rewrites the source IP of the packet to the NAT to allow for return traffic using NAT, then forwards the request onto the desired destination at 172.21.0.2. The request then works its way back to the real source, via the default gateway once more.

network_mode

docker compose supports another interesting property that we’ll take a quick look at: network_mode:. With this setting we can tell services to share the same network resources. As we’re using nginx in both services currently they will both try to bind on port 80 which will cause problems if they are sharing the same resources so for nginx_b we’ll create a unique nginx config file netsvc.conf:

server {
    listen       9080;
    server_name  localhost;
    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }
}

This is as simple an nginx config as you can get. Notably we’re telling it to listen on 9080 instead of the default 80 so that they can run in the same network namespace without colliding.

Now we’ll create an updated docker-compose-netsvc.yaml file:

---
version: '3.1'
services:
  nginx_a:
    image: 'nginx:latest'
    ports:
      - '8080:80'
      - '9080:9080'
  nginx_b:
    image: 'nginx:latest'
    volumes:
      - ./netsvc.conf:/etc/nginx/conf.d/default.conf:ro
    network_mode: "service:nginx_a"

See how we put our port-mapping for nginx_b (to :9080) in nginx_a? With this file in place we’ll run our new compose:

docker compose -f docker-compose-netsvc.yaml up -d
[+] Running 2/2
 ⠿ Container linuxnetworks-nginx_a-1  Started  0.4s
 ⠿ Container linuxnetworks-nginx_b-1  Started  0.6s

Running lsns this time we can see that we only have one network namespace (compared to two that we seen with the original docker-compose.yaml file):

sudo lsns -t net | grep nginx
4026533007 net      34 148278 root           0 /run/docker/netns/1e55105e0804 nginx: master process nginx -g daemon off;

We do still have two containers of course:

docker compose ps
NAME                      IMAGE               COMMAND                  SERVICE             CREATED             STATUS              PORTS
linuxnetworks-nginx_a-1   nginx:latest        "/docker-entrypoint.…"   nginx_a             5 minutes ago       Up 5 minutes        0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:9080->9080/tcp, :::9080->9080/tcp
linuxnetworks-nginx_b-1   nginx:latest        "/docker-entrypoint.…"   nginx_b             5 minutes ago       Up 5 minutes

We can see from docker inspect the nginx_b container has no visible network settings:

$ docker inspect $(docker ps -f 'Name=nginx_b' -q) -f '{{.NetworkSettings.Networks.linuxnetworks_default.IPAddress}}'
<no value>

$ docker inspect $(docker ps -f 'Name=nginx_b' -q) -f '{{.NetworkSettings.SandboxKey}}'

We can check the nginx_a network settings and see that all looks good there:

$ docker inspect $(docker ps -f 'Name=nginx_a' -q) -f '{{.NetworkSettings.Networks.linuxnetworks_default.IPAddress}}'
172.22.0.2
$ docker inspect $(docker ps -f 'Name=nginx_a' -q) -f '{{.NetworkSettings.SandboxKey}}'
/var/run/docker/netns/1e55105e0804

sudo nsenter --net=/var/run/docker/netns/1e55105e0804 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
80: eth0@if81: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:16:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0

You can imagine, say, running a VPN as a service and then forcing all other services to use that network namespace and resources. Super cool.

I hope this article has taught you a little about network namespaces and Dockers networking. Happy Hacking o/

A quick introduction to basic debugging tools for linux systems

Fri, 30 Dec 2022 14:28:12 +0000

The scope of this article is to cover the best common-use debugging tools available on Linux systems.

Firstly: let me qualify “basic”. I’m using this term because the breadth of debugging tools on Linux is so large. “basic” does not mean that these tools are super simple to understand deeply or if you aren’t already an expert in them that you are somehow “noob”; that’s not the case at all.

Secondly: there is no inventing being done here. I’m just surfacing information in this article which you can already find in the man pages. I would strongly recommend you check the relevant man page for the tool you find of interest in this article.

Thirdly: for the curious, these tools generally get their information from existing counters and statistics in /proc/ and /sys/. In the man pages for these tools you can see a FILES section which details the data source for the given tool.

“crisis tools”

This section is taken from Brendan Gregg’s book ‘System Performance’ (4.1.2). It covers not only where you can install some of the binaries mentioned in this article but also other useful tools for debugging. All credit to him for this list.

package	provides
procps	ps(1), vmstat(8), uptime(1), top(1)
util-linux	dmesg(1), lsblk(1), lscpu(1)
sysstat	iostat(1), mpstat(1), pidstat(1), sar(1)
iproute2	ip(8), ss(8), nstat(8), tc(8)
numactl	numastat(8)
linux-tools-common linux-tools-$(uname -r)	perf(1), turbostat(8)
bcc-tools (aka bpfcc-tools)	opensnoop(8), execsnoop(8), runqlat(8), runqlen(8), softirqs(8), hardirqs(8), ext4slower(8), ext4dist(8), biotop(8), biosnoop(8), biolatency(8), tcptop(8), tcplife(8), trace(8), argdist(8), funccount(8), stackcount(8), profile(8) and more …
bpftrace	bpftrace, basic versions of opensnoop(8), iolatency(8), iosnoop(8), bitesize(8), funccount(8), kprobe(8)
perf-tools-unstable	Ftrace versions of opensnoop(8), execsnoop(8), iolatency(8), iosnoop(8), bitesize(8), funccount(8), kprobe(8)
trace-cmd	trace-cmd(1)
nicstat	nicstat(1)
ethtool	ethtool(8)
tiptop	tiptop(1)
msr-tools	rdmsr(8), wrmsr(8)
github.com/brendangregg/msr-cloud-tools	showboost(8), cpuhot(8), cputemp(8)
github.com/brendangregg/pmc-cloud-tools	pmcarch(8), cpucache(8), lcache(8), tlbstat(8), resstalls(8)

top

When: A reasonable first place to look for an issue.

This is a command-line utility in Linux that allows users to view the processes running on their system and monitor their resource usage in real-time. It can be used to identify performance bottlenecks, track the usage of system resources, and troubleshoot issues on a Linux system.

The output of top will include a list of processes running on the system, along with the following information for each process:

The process ID (PID)
The user owning the process
The CPU and memory usage of the process
The command that started the process

From the output you can identify some performance issues with your system. For example, if a single process is consuming a large amount of CPU or memory resources, it could be causing performance issues for your system. Similarly, if multiple processes are consuming high amounts of resources, it could indicate that your system is struggling to keep up with demand and may be a bottleneck.

One common problem that top can help identify is resource contention which occurs when multiple processes or applications are competing for the same system resources.

I suspect this tool is familiar to many of you already:

However what you may not realise is that you can pull much more data from it. Press ‘f’ when in top to access the fields management view and select other columns to display:

htop is a visually prettier alternative to top and that can make understanding the system at a glance easier.

mpstat

When: you think there’s an issue with the CPU utilization.

mpstat is a command-line tool that is used to monitor the performance of a Linux system by displaying the CPU usage of each processor and the overall system.

Imagine that you are the system administrator of a web server that is experiencing slow performance and perhaps even downtime. You check the web server logs but nothing obvious is standing out so you begin to suspect there’s something else wrong with the system. One general check we can do is on the CPU cores - this is where mpstat comes in.

mpstat -P ALL

This will display the CPU usage statistics for all CPU cores on the system. The output will look something like this:

Linux 4.9.0-8-amd64 (server.petermcconnell.com)  01/01/2023  _x86_64_    (2 CPU)

02:34:56 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:34:56 AM  all    20.00    0.00    5.00    0.00    0.00    0.00    0.00    0.00    0.00   75.00
02:34:56 AM    0    22.50    0.00    6.25    0.00    0.00    0.00    0.00    0.00    0.00   71.25
02:34:56 AM    1     7.50    0.00    3.75    0.00    0.00    0.00    0.00    0.00    0.00   88.75

The first line shows the Linux version, hostname, and CPU architecture. The second line displays the column headers, which show the various metrics that are being measured. The third line, labeled “all,” shows the overall CPU usage for all CPU cores combined. The fourth and fifth lines show the CPU usage for each individual CPU core.

In this example, we can see that CPU 0 is running at a higher utilization than CPU 1. This could indicate that there is an issue with CPU 0 that is causing it to work harder than it should. To further investigate the issue we can opt to print these statistics every second by adding a “1” to the end of the command:

mpstat -P ALL 1

This will print statistics from each core every second, showing the current CPU usage. By watching the output over time, we can see if there are any patterns or spikes in the usage of a particular CPU core.

If we notice that one of the CPU cores is consistently running at a higher utilization than the others, we can try to identify the cause of the problem. One way to do this is by using the top command to see which processes are using the most CPU resources. It could be that the program itself is misconfigured and not properly distributing load over the available cores.

iostat

When: you think there’s an issue with IO.

The iostat tool is a command-line utility in Linux that allows users to monitor input/output (I/O) statistics for devices, partitions, and network filesystems. It can be used to identify performance bottlenecks and track the usage of system resources by I/O operations.

To use iostat, you need to specify the interval at which you want to collect data, as well as the devices or partitions you want to monitor. For example, the following command will display I/O statistics for all devices every 2 seconds, 5 times:

$ iostat -p ALL 2 5
Linux 5.4.0-72-generic (myputer) 	30/12/22 	_x86_64_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.40   15.48    4.48    0.99    0.00   77.66

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
loop0             0.00         0.00         0.00         0.00          0          0          0
loop1             0.00         0.00         0.00         0.00          0          0          0
loop2             0.00         0.00         0.00         0.00          0          0          0
loop3             0.00         0.00         0.00         0.00          0          0          0
loop4             0.00         0.00         0.00         0.00          0          0          0
loop5             0.00         0.00         0.00         0.00          0          0          0
loop6             0.00         0.00         0.00         0.00          0          0          0
loop7             0.00         0.00         0.00         0.00          0          0          0
sda               4.83        79.27       106.76         0.00     858983    1156820          0

... repeat

You can also get more simple summaries - for example, CPU utilization report:

$ iostat -c
Linux 5.4.0-72-generic (urputer) 	30/12/22 	_x86_64_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.36   15.09    4.37    0.95    0.00   78.22

Or a device utilization report:

iostat -d -h -p ALL
Linux 5.4.0-72-generic (xerxes) 	30/12/22 	_x86_64_	(4 CPU)

      tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd Device
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop0
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop1
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop2
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop3
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop4
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop5
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop6
     0.00         0.0k         0.0k         0.0k       0.0k       0.0k       0.0k loop7
     4.82        73.5k       103.6k         0.0k     841.2M       1.2G       0.0k sda
     0.01         0.3k         0.0k         0.0k       3.1M       0.5k       0.0k sda1
     0.00         0.0k         0.0k         0.0k       2.0k       0.0k       0.0k sda2
     4.71        73.1k       103.6k         0.0k     836.0M       1.2G       0.0k sda5

We’re seeing a lot of devices here which are showing to be inactive. We can easily hide those from view - rarely when debugging an issue do you want to see statistics for a device that isn’t being used. To hide these, pass -z:

iostat -d -h -p ALL -z
Linux 5.4.0-72-generic (xerxes) 	30/12/22 	_x86_64_	(4 CPU)

      tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd Device
     4.84        72.1k       103.1k         0.0k     841.2M       1.2G       0.0k sda
     0.01         0.3k         0.0k         0.0k       3.1M       0.5k       0.0k sda1
     0.00         0.0k         0.0k         0.0k       2.0k       0.0k       0.0k sda2
     4.73        71.7k       103.1k         0.0k     836.0M       1.2G       0.0k sda5

By analyzing this data, you can identify performance issues with your devices or partitions. For example, if the number of read or write operations is consistently high, it could indicate that the device or partition is heavily used and may be a bottleneck. Similarly, if the average time taken for each I/O operation is consistently high, it could indicate that the device or partition is struggling to keep up with demand.

A common problem that iostat can help identify is disk saturation, which occurs when a disk is being used to its maximum capacity and can’t keep up with the demand for I/O operations. This can lead to slow performance and potential data loss. In iostat output this problem may manifest as a consistently high percentage of time the device is busy with I/O operations and a high average time taken for each I/O operation.

sar

When: general purpose. you think there’s an issue with CPU, Memory, Network or Block devices. Can record findings to files on disk for comparisson later (which is the main selling point of sar over, say vmstat)

The sar (System Activity Report) tool is a command-line utility in Linux that allows users to monitor various system performance metrics over time. It can be used to identify performance bottlenecks, track the usage of system resources, and troubleshoot issues on a Linux system.

To use sar, you need to specify the interval at which you want to collect data and the system performance metrics you want to monitor. For example, the following command will display CPU utilization statistics every 2 seconds:

sar 2

The output of sar will include various performance metrics, depending on the options you specify. Some of the metrics that sar can monitor include:

CPU utilization
Memory usage
Disk I/O activity
Network activity
Load average (a measure of the system’s CPU and I/O utilization)

By analyzing this data, you can identify any performance issues with your system. For example, if the CPU utilization is consistently high, it could indicate that your system is struggling to keep up with demand and may be a bottleneck. Similarly, if the memory usage is consistently high, it could indicate that your system is running out of available memory.

sar stores its reports in /var/log/sysstat/ by default and will look for files in this directory when you ask it for some reports. The file it looks for changes depending on the INTERVAL value passed to it. This can result in an error if a report isn’t available from yesterday:

# -B tells sar to create a report on paging statistics
sar -B
Cannot open /var/log/sysstat/sa30: No such file or directory
Please check if data collecting is enabled

You can specify 0 as the interval time which tells sar to use statistics for the time since the system was started:

sar 0 -B
Linux 5.4.0-72-generic (herputer) 	30/12/22 	_x86_64_	(4 CPU)

17:51:10     pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
17:51:10        66.01     99.20   3074.99      0.38   1853.06      0.00      0.00      0.00      0.00

sar can also be used to report on I/O and transfer rate statistics:

sar 0 -b
Linux 5.4.0-72-generic (hisputer) 	30/12/22 	_x86_64_	(4 CPU)

17:53:49          tps      rtps      wtps      dtps   bread/s   bwrtn/s   bdscd/s
17:53:49         4.64      1.32      3.32      0.00    130.43    196.40      0.00

… and block devices:

sar 0 -d
Linux 5.4.0-72-generic (ourputer) 	30/12/22 	_x86_64_	(4 CPU)

17:55:12          DEV       tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util
17:55:12       dev8-0      4.63     64.81     97.66      0.00     35.13      0.05     11.69      2.11

… and more. sar is a very powerful tool - please check out the man page for more info.

vmstat

When: you think there’s an issue with memory usage or I/O activity.

The vmstat (Virtual Memory Statistics) tool is a command-line utility in Linux that allows users to monitor various system performance metrics, including memory usage and I/O activity. It can be used to identify performance bottlenecks, track the usage of system resources, and troubleshoot issues on a Linux system.

To use vmstat you need to specify the interval at which you want to collect data. For example, the following command will display system performance statistics every 2 seconds:

vmstat 2

The output of vmstat will include the following information:

The number of processes in various states (e.g. running, waiting, blocked)
The amount of memory being used, including the amount of available memory
The number of page faults (when a process requests a page of memory that is not in physical memory and must be retrieved from disk)
The amount of I/O activity, including the number of read and write operations per second

free

When: you want to assess the state of memory on a machine.

free -w -h
              total        used        free      shared     buffers       cache   available
Mem:           15Gi       1.1Gi        12Gi       224Mi        95Mi       1.4Gi        13Gi
Swap:         2.0Gi          0B       2.0Gi

Here we see:

The total amount of available and used memory
The amount of used and available swap space (virtual memory on disk used to store data when the system’s physical memory is full)
The number of used and available buffers (a type of cache used to store data temporarily) and cache (data stored in memory to speed up access to frequently used data)

lsblk

When: you assume there’s an issue with the block devices and want to first check everything is in place.

The lsblk (List Block Devices) command is a command-line utility in Linux that allows users to view a list of block devices (e.g. hard drives, SSDs, USB drives) attached to their system. It can be used to view the available block devices, their mount points, and their partition layouts. You may want to use this tool to validate if block devices are available and mounted.

To use lsblk, you simply need to enter the command followed by any desired options. The following command will display a list of all block devices on the system:

$ lsblk -f
NAME   FSTYPE LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1 vfat         29C1-1F85                               511M     0% /boot/efi
├─sda2
└─sda5 ext4         e3a6ec94-05ab-4e15-a310-e5797a2c55e9    419G     3% /

The output of lsblk will include the following information for each block device:

The device name (e.g. sda, sdb)
The device type (e.g. disk, partition)
The size of the device
The mount point (if the device is mounted)
The filesystem type (if the device is formatted with a filesystem)

df

When: you’re aware of disk space issues and want to check the current availability.

The df (Disk Free) command is a command-line utility in Linux that allows users to view the amount of available and used disk space on their system. It can be used to identify if a disk is approaching or at capacity.

To use df, you simply need to enter the command followed by any desired options. The following command will display the total amount of available and used disk space on the system:

df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       457G   15G  420G   4% /

The output of df will include the following information for each file system:

The file system name (e.g. /dev/sda1)
The total size of the file system
The amount of used and available space on the file system
The percentage of used space on the file system
The mount point (the location where the file system is mounted in the directory structure)

perf

When: you are aware of a processing issue and want to understand which process it’s coming from, or you want to drill into the process to understand why it’s not performing to the expected standard.

This tool allows users to monitor various performance metrics and events on their system. It can be used to identify performance bottlenecks, track the usage of system resources, and troubleshoot issues on a Linux system. Unlike the other tools in this article perf allows us to actually peek into the running program to see which parts of it are consuming cpu cycles. This is an incredibly valuable tool to have for those wanting to performance engineer on Linux and I have a practical example of this detailed in https://www.petermcconnell.com/posts/perf_eng_with_py12/.

To use perf, you need to specify the type of performance data you want to collect, as well as the specific events or metrics you want to monitor. To get general cpu profiling you can run:

perf top -a

Perf also allows you to gather performance statistics. For example, the following command will collect CPU performance data for the cpu-clock event:

perf stat -e cpu-clock
^C
 Performance counter stats for 'system wide':

         11,633.97 msec cpu-clock                 #    3.998 CPUs utilized

       2.909972618 seconds time elapsed

The scope of what you can do with perf is huge - too large for this article, but I couldn’t leave it off the list. Perhaps a perf deep dive article will come here soon …

uptime

When: you want to get a quick understanding of system load over the past 1, 5, 15 minutes or want to see how long the machine has been up for.

uptime
19:26:35 up  5:12,  1 user,  load average: 0.33, 0.37, 0.37

Perf engineering with Python 3.12

Mon, 26 Dec 2022 22:54:29 +0000

overview

3.12 brings perf profiling! In this article we take a look at how the new perf profiling support helps reduce our dummy Python script from 36 seconds to 0.8 seconds. We’ll introduce the Linux tool perf and also FlameGraph.pl, look at some disassembly and go bug hunting. You can view the code for this article here: https://github.com/peter-mcconnell/petermcconnell.com/tree/main/assets/dummy/perf_py_proj

Take a second to go check out https://docs.python.org/3.12/howto/perf_profiling.html and indeed the changelog at https://www.python.org/downloads/release/python-3120a3/. The important part (for this post) from these links is:

The Linux perf profiler is a very powerful tool that allows you to profile and obtain information about the performance of your application. perf also has a very vibrant ecosystem of tools that aid with the analysis of the data that it produces.

The main problem with using the perf profiler with Python applications is that perf only allows to get information about native symbols, this is, the names of the functions and procedures written in C. This means that the names and file names of the Python functions in your code will not appear in the output of the perf.

Since Python 3.12, the interpreter can run in a special mode that allows Python functions to appear in the output of the perf profiler. When this mode is enabled, the interpreter will interpose a small piece of code compiled on the fly before the execution of every Python function and it will teach perf the relationship between this piece of code and the associated Python function using perf map files.

writing a “bad” program

I’m excited to try this, so lets get going. Firstly lets create a python script for us to profile. I’m doing this before installing Python 3.12 as I want to create a FlameGraph of how this process looks in 3.10 verses 3.12. Here we have a script that attempts to perform lookups against a large list:

import time


def run_dummy(numbers):
    for findme in range(100000):
        if findme in numbers:
            print("found", findme)
        else:
            print("missed", findme)


if __name__ == "__main__":
    # create a large sized input to show off inefficiency
    numbers = [i for i in range(20000000)]

    start_time = time.time()  # get the current time [start]
    run_dummy(numbers)  # run our inefficient method
    end_time = time.time()  # get the current time [end]

    duration = end_time - start_time  # Calculate the duration
    print(f"Duration: {duration} seconds")  # Print the duration

Running this I get the following result:

python3.10 assets/dummy/perf_py_proj/before.py
...
found 99992
found 99993
found 99994
found 99995
found 99996
found 99997
found 99998
found 99999
Duration: 36.06884431838989 seconds

36 seconds is bad enough for us to pick up a reasonable amount of samples.

flamegraphs!

Now we can create our FlameGraph:

# record profile to "perf.data" file (default output)
perf record -F 99 -g -- python3.10 assets/dummy/perf_py_proj/before.py
# read perf.data (created above) and display trace output
perf script > out.perf
# fold stack samples into single lines
# here I reference ~/FlameGraph/ - you can get this from https://github.com/brendangregg/FlameGraph
~/FlameGraph/stackcollapse-perf.pl out.perf > out.folded
# generate flamegraph
~/FlameGraph/flamegraph.pl out.folded > ./assets/perf_example_python3.10.svg

This gives us a nice SVG that visualises the traces:

This isn’t useful … I can see most of the time was spent in “new_keys_object.lto_priv.0” but that is meaningless in the context of the code.

time for Python 3.12…

First I need to install it - the steps for this vary depending on OS - follow the build instructions here for your environment: https://github.com/python/cpython/tree/v3.12.0a3#build-instructions

# for me on ubuntu:22.04
# ensure I have python3-dbg installed
sudo apt-get install python3-dbg

# build python
export CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"
./configure --enable-optimizations
make
make test
sudo make install
unset CFLAGS

# after this I reset my systems python3 symlink to 3.10 as 3.12 isn't yet stable
# for testing python3.12 I'll call "python3.12" instead of "python3"
ln -sf /usr/local/bin/python3.10 /usr/local/bin/python3

With that installed I first need to enable perf support. This is detailed in https://docs.python.org/3.12/howto/perf_profiling.html and there are three options: 1) an environment variable, 2) an -X option or 3) dynamically using sys. I’ll go for the environment variable approach as I don’t mind everything being profiled for a small script:

export PYTHONPERFSUPPORT=1

Now we simply repeat the process above using the python3.12 binary instead:

# record profile to "perf.data" file (default output)
perf record -F 99 -g -- python3.12 assets/dummy/perf_py_proj/before.py
# read perf.data (created above) and display trace output
perf script > out.perf
# fold stack samples into single lines
# here I reference ~/FlameGraph/ - you can get this from https://github.com/brendangregg/FlameGraph
~/FlameGraph/stackcollapse-perf.pl out.perf > out.folded
# generate flamegraph
~/FlameGraph/flamegraph.pl out.folded > ./assets/perf_example_python3.12.before.svg

First we’ll take a peek at the report with perf report -g -i perf.data

Awesome! We can see our Python function names and script names!

Now we can take a look at the updated SVG that visualises the traces with Python 3.12:

This is already looking much more useful. We see the majority of the time is being spent doing comparisons and in the list_contains method. We can also see the specific file before.py and method run_dummy that is calling it.

investigation time / the fix

Now that we know where in our code the problem is, we can take a look at the source code in CPython to see why the list_contains method would be so slow: https://github.com/python/cpython/blob/199507b81a302ea19f93593965b1e5088195a6c5/Objects/listobject.c#L440

note: you may not always have access to the source code - in circumstances such as this you can view the disassembly in perf report directly to get some idea of what’s going on. I’ll add a quick section at the end showing how this looks

// I found this by going to https://github.com/python/cpython/ and searching for "list_contains"

static int
list_contains(PyListObject *a, PyObject *el)
{
    PyObject *item;
    Py_ssize_t i;
    int cmp;

    for (i = 0, cmp = 0 ; cmp == 0 && i < Py_SIZE(a); ++i) {
        item = PyList_GET_ITEM(a, i);
        Py_INCREF(item);
        cmp = PyObject_RichCompareBool(item, el, Py_EQ);
        Py_DECREF(item);
    }
    return cmp;
}

Nasty… looking at this code I can see that each time it is invoked it iterates over the array and performs a comparison against each item. That’s far from ideal for our usecase, so lets go back to the Python code we wrote. Our Flamegraph shows us that the problem is in our run_dummy method:

def run_dummy(numbers):
    for findme in range(100000):
        if findme in numbers:  #  <- this is what triggers list_contains
            print("found", findme)
        else:
            print("missed", findme)

We can’t really change that line as it is doing what we want it to do - identifying if an integer is in numbers. Perhaps we can change the numbers data type to one better suited to lookups. In our existing code we have:

    numbers = [i for i in range(20000000)]

    start_time = time.time()  # get the current time [start]
    run_dummy(numbers)  # run our inefficient method

Here we used a LIST data type for our “numbers”, which under the hood (in CPython) is implemented as dynamically-sized arrays and as such are nowhere near as efficient (O(N)) as the likes of a Hashtable for looking up an item (which is O(1)). A SET on the other hand (another Python data type) is implemented as a Hashtable and would give us the fast lookup we’re looking for. Lets change the data type in our Python code and see what the impact is:

    # we'll just change this line, casting numbers to a set before running run_dummy
    run_dummy(set(numbers))  # passing a set() for fast lookups

Now we can repeat the steps as above to generate our new flamegraph:

# record profile to "perf.data" file (default output)
perf record -F 99 -g -- python3.12 assets/dummy/perf_py_proj/after.py
...
found 99998
found 99999
Duration: 0.8350753784179688 seconds
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.039 MB perf.data (134 samples) ]

Already we can see that things have massively improved. Where previously this was taking 36 seconds to run it is now taking 0.8 seconds! Lets continue creating our flamegraph for the new improved code:

# read perf.data (created above) and display trace output
perf script > out.perf
# fold stack samples into single lines
# here I reference ~/FlameGraph/ - you can get this from https://github.com/brendangregg/FlameGraph
~/FlameGraph/stackcollapse-perf.pl out.perf > out.folded
# generate flamegraph
~/FlameGraph/flamegraph.pl out.folded > ./assets/perf_example_python3.12.after.svg

This is a much healthier looking Flamegraph and our application is now much faster as a result. The perf profiling support in Python 3.12 brings a tremendously useful tool to software engineers that want to deliver fast programs and I’m excited to see the impact this will have on the language.

bonus round: what to do when you can’t access the source code?

Sometimes you don’t have access to the underlying code which can make trying to understand what’s going on much more difficult. Thankfully perf report allows us to view the dissassembled code which can help paint a picture of what the machine is actually doing. This is a reasonable first place to look - I tend to prefer the source code if I can get hold of it as it allows me to git blame / view the associated commits and PRs. To view this you can do the following:

Open the perf report and select the line we’re interested in:

# this assumes we have already ran 'perf record' to generate perf.data ...
perf report -g -i perf.data

Press enter and choose the annotate option:

Behold! Here we can see both the C code and the machine instructions. Super useful! You can compare the screenshot below against the code snippet we’re interested in: https://github.com/python/cpython/blob/199507b81a302ea19f93593965b1e5088195a6c5/Objects/listobject.c#L440

Current Reading

Tue, 29 Nov 2022 20:30:52 +0000

The Linux Programming Interface

I should have read this years ago. It’s a big boi, but if you work around linux or are interested in it - this book is a critical primer on many of the core topics you’ll need to really understand what’s going on. I binged this book on the first pass over multiple weeks - deliberately skipping implementing the code examples, opting to just read them / ensure I understood them. I choose to do this so that I could get over as much content as possible, with the view of circling back over the examples once I’m finished The C Programming Language book so that I can make better decisions about the code I’m writing.

The C Programming Language

I generally try to avoid taking on multiple books but I figured as the above is so large and much of the books example code is given in C, then this book which I’m long overdue on would be a nice complement.

It’s fine. Concise and clear wording; unlike the print quality, unfortunately. This books print looks like it was hand-painted by a very nervous individual trying to hit a deadline in the dark. Print-quality aside, I’d say this is an important book to read, but not much of a page-turner. For me the biggest advantage of the book was seeing syscall libraries in action and also appreciating more about what cpp gives us.

Up next

I have two “crackers” on my desk. Really looking forward to these:

Linux Kernel Development (https://amzn.to/3hXuWTw)
Systems Performance (https://amzn.to/3I9iU49)