A Container Is a Posture, Not a Primitive

There is no create_container() syscall. The kernel will not hand you a container, because the kernel has no idea what one is. What it has is a pile of unrelated primitives — clone with a fistful of flags, pivot_root, a cgroup directory, a capability bounding set, a seccomp filter, mount propagation rules — and a "container" is what you get when you arrange all of them, in the right order, in the brief window before you call execve and hand the process to the user's program.

That is the whole thesis of this series, and it is worth saying plainly because it is the thing the abstraction hides: a container is a posture, not a primitive. It is a stance a process is forced into. Docker did not discover a container object inside Linux; it discovered an order of operations. Every isolation property you associate with the word — its own process tree, its own root filesystem, its own hostname, a memory ceiling, a network of its own — is a separate kernel mechanism with its own setup call, its own failure modes, and, as I learned repeatedly, its own way of quietly contradicting whatever mental model you walked in with.

I wanted to understand that order of operations the only way I trust to actually learn something, which is to build it. So I wrote minicontainer: a container runtime in C, from fork/execve up to OCI bundles, where each isolation boundary is introduced on its own, in its own phase, with nothing else in the way. This post is the map of that build, and the contract for the rest of the series.

Why C, and why one boundary at a time

A Go implementation — and the canonical "containers from scratch" talks are all in Go — is a better way to ship a runtime and a worse way to learn one. Go's runtime is a co-author you didn't ask for. It spawns threads behind your back (which matters enormously the moment you touch CLONE_NEWUSER and setns), it manages file descriptors for you, and it papers over the exact syscall-level sharp edges this series is about. In C with raw syscalls there is no co-author. When setns(2) returns EINVAL, it is returning it to you, about a decision you made, and there is nowhere to hide from the reason.

The second constraint is the structure: one isolation concern per phase. Phase 1 adds the PID namespace and only the PID namespace. Phase 2 adds the mount namespace and pivot_root. Each phase introduces a new module that supersedes the previous phase's top-level execution function while leaving the old one in the tree as a regression baseline — so the thing always builds, always passes its earlier tests, and never collapses into a single 2,000-line main.c that you have to reverse-engineer to understand any one decision.

The payoff of that discipline is the part I care most about: every boundary is a git tag. There are fifteen of them, phase-0 through phase-8c. You can git checkout phase-4b and the runtime in front of you is exactly a fork/exec baseline plus PID, mount, overlay, UTS, and user namespaces — and not one line of the cgroup, network, or hardening code that came later. The complexity at any tag is precisely the complexity that boundary requires. Almost no "from scratch" writeup gives you that; they give you a finished file and ask you to imagine the path.

One caveat, and it turns out to be the more interesting framing rather than a disclaimer: a tag preserves the runtime exactly as it was at that phase — including the bugs I had not found yet. The cgroup limits I added at phase-5 did not actually enforce anything until a fix two phases later: the child was placed into its cgroup a moment after it had already been let loose, and every cgroup test I'd written checked that the limit file held the right value — never that a fork was actually denied — so nothing caught it until a live run at phase-7b. So these fifteen tags are not snapshots of correct code. They are snapshots of what I believed was correct at the time — which is the more honest thing to preserve, and untangling which of those beliefs were wrong is most of what the next post is about.

The map

Here is the whole build. Each row is a tag you can check out.

Read top to bottom and you are watching a process launcher grow, one veto at a time, into something that can run an Alpine bundle built by somebody else's tooling. Nothing in that list is magic. All of it is an arrangement of primitives the kernel already had.

The two things that make this worth your time

There are a lot of "I built a container" posts, and most of them are fine, and you have probably read the good ones (Liz Rice's Containers From Scratch is the canonical live-coding version; runc and youki are the production answers, and they are open source). I am not going to out-survey any of them, so I won't try. What this series has that a survey doesn't is two things.

The first is the checkout-able history above. The posts that follow are not illustrated with hopeful pseudocode; they are illustrated with a tag and a path, and you can go look.

The second is the failure trail. The repo carries a docs/decisions.md with thirty-seven numbered architecture decisions and twenty-seven numbered errors — every error written up with its symptom, its root cause, and the commit that fixed it. These are not "oops, typo" entries. They are the places where the kernel told me my model was wrong: a bind mount that mounted nothing because I ran it after pivot_root had already detached the source; a setns that returned EINVAL because I tried to join a namespace I was already in. The single best post in this series is built entirely out of those, and it exists because I wrote the failures down while they were still embarrassing.

The contract: I would rather you verify than trust me

This series is written for people who will fact-check it, so let me make that easy and make it a promise up front. Every technical claim in every post is anchored to at least one of these:

• A tag and a path. "git checkout phase-8b, then src/hardening.c." The tag matters: line numbers drift between phases, so a claim is only valid at the tag it names — never against the moving tip of master. The repo is public (link at the bottom).
• A decisions.md number. When I say a cgroup limit was silently unenforced — as I did above — the anchor is "Error #25," and you can read the entry yourself: symptom, root cause, the fixing commit, and confirmation that it was latent from Phase 5 and only caught at Phase 7b.
• A command you can paste. Where a claim is about runtime behavior, you get the invocation and the expected output — including, honestly, the environment it needs (root, cgroup v2, x86-64), because several of these behaviors are environment-sensitive in ways that are themselves part of the story.
• An external authority. For kernel behavior the repo can't prove on its own — man 7 user_namespaces, man 2 seccomp, the OCI Runtime Spec, a CVE number.

If a sentence asserts something about how Linux behaves and can't point to one of those four, it does not belong in the post. That is the standard the whole series is held to, and you should hold it to that standard.

What this is not

minicontainer is a teaching artifact, and the honest framing of its limits is part of the teaching. It is not a runc replacement and it is not trying to be. The rootfs is built by hand rather than pulled from a registry (the pull subcommand in Phase 8c is a hardcoded two-image whitelist, not a registry client — docs/decisions.md says so in as many words). The OCI parser understands about twenty-two of the spec's roughly one hundred and twenty fields, and the ones it ignores are listed, on purpose. There are documented limitations — the docs/decisions.md "Known Limitations" section is its own list — and where a hardening layer is weaker than Docker's by one defense, I say which one and why I made that call rather than quietly closing the gap. Naming the non-goals is not modesty; for this audience it is the difference between a credible writeup and a demo.

What's coming

The rest of the series, in rough order:

• The war stories. The flagship. Six or so places where the Linux kernel refused to honor an assumption I'd have sworn was safe — fds cross namespaces but paths don't; a bind mount after pivot_root mounts nothing; setns into your own user namespace is an error; a resource limit that was a function of timing, not value; a --secure flag that was dead on arrival because validation ran eighteen lines before the field it validated was set. Each one is short, each one has a tag and an error number, and each one taught me something a working example never would have.
• Namespaces and the rootless wall. What PID, mount, UTS, user, and IPC namespaces each actually buy you, why pivot_root beats chroot, how the UID/GID map makes rootless work — and the afternoon an Ubuntu 24.04 AppArmor profile refused a rootless sethostname the user namespace entitled me to make, the same binary failing from an SSH session and succeeding, absurdly, from the editor's integrated terminal.
• cgroups v2, and the limit that did nothing. Wiring memory/CPU/PID limits by hand, and the ordering bug that made them silent no-ops until you thought to watch memory.current.
• Hardening with no libraries. Dropping to Docker's fourteen-capability profile, NO_NEW_PRIVS, and a seccomp allow-list, all hand-assembled with raw syscalls and zero libcap/libseccomp — including why clone3 has to return ENOSYS specifically or glibc's threading breaks, and a place where two defense layers deliberately disagree.
• Deleting 95% of five functions. A non-kernel post about engineering judgment: when not to abstract (and why I let the duplication sit for five phases on purpose), and what made it safe to finally collapse it.
• OCI bundles and just enough of the spec. A hand-rolled recursive-descent JSON parser, the parsed-versus-ignored field table, and the security call that bundles are trusted input.

The repo

minicontainer lives at github.com/Enginerd-2019/minicontainer. Clone it with --recursive (the /proc-reading code is a submodule), and the phase tags are all there: git tag -l 'phase-*' lists fifteen, phase-0 through phase-8c. docs/decisions.md is the thirty-seven-decision, twenty-seven-error logbook that the rest of this series quarries. Building needs nothing exotic — a Linux box and GCC, warning-clean under -Wall -Wextra; the full test suite needs root and a controlling TTY (there's a documented reason for the TTY, and it is itself one of the war stories).

If you want to follow along rather than read along, the move is: check out phase-0, read the two short source files that launch a process (src/main.c and src/spawn.c, about 250 lines together), and then walk the tags forward. Each one adds exactly one reason the process is more contained than it was at the tag before. By phase-8c you have assembled the whole posture — and, more usefully, you know exactly when in the sequence each part of it goes on, and what happens when you get the order wrong.

There is no create_container() syscall. There is only the order you do things in, and this is the series about getting that order right by first getting it comprehensively wrong.