Docker Is Weird

I decided to create my own container runtime out of curiosity and, even though I'm still in the early stages, I've discovered some peculiar aspects of Docker. Unlike virtual machines, containers are merely processes with specific limitations. You still have the kernel running that process, and if you know where to look, you can find it in /proc.

Docker isn't a single Linux feature either; instead, it's a combination of five different kernel features. Namespaces hide what the host and container can access, cgroups limit resource usage within the container, and there's complex networking magic that isolates a container's network. Images, however, are just directories that Docker places somewhere and then uses chroot (or pivot_root) to make that directory the container's new root.

When you run Docker's container runtime, runc, with the command "docker run -it --rm ubuntu /bin/bash," it performs several actions. First, runc, written in Go, creates a new process. Then, it runs the chroot or pivot_root command to set up the container's new root directory. Go, the language runc is written in, has some design choices that make it less ideal for a container runtime.

It is multithreaded, which is a great feature, but it presents challenges in runc. Many of the Linux syscalls runc needs to make must be done in a single-threaded environment, so runc has a complex C module that runs before Go's runtime initializes to handle those syscalls. Go also doesn't have a fork function, which is crucial for a container runtime.

To compensate, runc executes /proc/self/exe with a special command-line argument that identifies it as a child process, mimicking a fork. This workaround can be easily overlooked by those, like me, who use Docker on macOS. Many Linux kernel APIs container runtimes rely on require root privileges, so the Docker daemon typically runs as root.

Container security is undoubtedly a challenging task. However, this has improved over time, with networking being the main exception. When running rootless container runtimes, they seem to have to rebuild the entire networking stack in userspace to provide the same features as Docker. As I'm just starting my own container runtime, I've already learned so much about Docker by writing my own version of a tool I use daily.