A namespace is a lie the kernel tells one process about the shape of the system. Inside a PID namespace, the process is PID 1 and the host's other ten thousand processes don't exist. Inside a mount namespace, the mount table is its own copy, and pivot_root can swap the root out from under it without the host noticing. Inside a UTS namespace, hostname returns whatever you told it to. The lies are independent — you can tell any subset of them — and each one is a single bit in the flags argument to clone(2).
That independence is the whole design of the namespace phases of minicontainer — phase-1, phase-2, phase-4, phase-4b, phase-4c. Each of them adds one bit, one boundary, and (usually) one helper the child calls after clone returns. This post walks the five namespaces the runtime uses by phase-4c — PID, mount, UTS, IPC, user — roughly in the order they were built, with the bugs each one produced, saving for last the one that is different in kind: the user namespace, which makes the whole thing run without sudo, right up until the host's security policy quietly decides otherwise.
As with the rest of this series, every claim carries a tag, a docs/decisions.md number, or a man page. The anchors are collected at the end.
The flags, and the rules that set them
Here is the entire isolation surface at phase-4c, as the parent assembles it before the one clone call — condensed from phase-4c:src/uts.c::uts_exec, with the debug prints stripped out but the order preserved:
int flags = SIGCHLD;
if (config->enable_user_namespace) flags |= CLONE_NEWUSER;
if (config->enable_pid_namespace) flags |= CLONE_NEWPID;
if (config->enable_mount_namespace) flags |= CLONE_NEWNS;
if (config->enable_uts_namespace) flags |= CLONE_NEWUTS;
if (config->enable_ipc_namespace) flags |= CLONE_NEWIPC;
Five bits. Everything else — the pivot, the hostname, the UID map — is work the parent or child does after the namespaces exist. And the bits are not all set directly by the user. Three CLI invariants — the mount rule from Phase 2, the PID auto-enable from Phase 3, the UTS rule from Phase 4 — sit together at phase-4b:src/main.c:228-230 and survive unchanged into phase-7b:src/cli.c (whitespace aligned here):
.enable_pid_namespace = enable_pid_namespace || (rootfs_path != NULL),
.enable_mount_namespace = (rootfs_path != NULL),
.enable_uts_namespace = (hostname != NULL),
--rootfs auto-enables both the mount namespace (pivot_root works on whatever mount namespace you are in — without a fresh one it would swap the host's root) and the PID namespace (a private root with the host's process table visible through /proc is a confusing half-measure). --hostname implies UTS. The user asks for outcomes; the flags follow. This matters later, because it means "I passed the right flags" is a statement the runtime can guarantee — and the veto at the end of this post happens anyway.
PID: your own process one
CLONE_NEWPID is the first bit (phase-1), and it is the one people mean when they say "container." The child becomes PID 1 of a fresh namespace; its own getpid() returns 1; processes it spawns get 2, 3, 4; and from inside, the host's processes are not hidden so much as unaddressable — there is no PID that names them.
There is a catch that every first implementation hits: creating the namespace does not change what /proc shows. /proc is a filesystem, and the one mounted at boot reflects the namespace it was mounted from — the host's. Run ps in a fresh PID namespace with the host's /proc still mounted and you see every host process, just with PIDs that don't resolve. The fix is to mount a fresh proc instance after entering the namespace, which is exactly why mount_proc runs in the child after the rootfs is in place (man 7 pid_namespaces, "/proc and PID namespaces"). It is also why the mount namespace comes next: without one, that fresh proc would land in the host's mount table.
Mount: pivot_root, and the two errors it took to get there
CLONE_NEWNS gives the child a copy of the parent's mount table that it can rearrange freely. The interesting decision is what to rearrange it with.
The tempting answer is chroot. It's one call, it has a glibc wrapper, and it is not a container primitive. The first reason Decision #10 gives for rejecting it is escapes: a process that holds an open fd to a directory outside the chroot can fchdir to it; a process whose working directory is outside can .. its way up; and a second chroot from inside can slip the first one. chroot changes the apparent root and leaves every reference to the real one intact. pivot_root actually swaps the root mount, and once the old root is unmounted there is nothing left to escape to. It's what runc, crun, and Docker use, and — the property I came to appreciate most — the kernel refuses to do it unless the mount tree is set up properly. chroot will happily let you build something broken.
pivot_root has no glibc wrapper (syscall(SYS_pivot_root, ...)), and the correct dance is five steps, as Decision #11 lists them — the four mount/umount2 calls verbatim from phase-2:src/mount.c::setup_rootfs, step 4 condensed (the source names old_root through a put_old variable and passes mode 0700 to mkdir):
mount("", "/", NULL, MS_PRIVATE | MS_REC, NULL)— make the inherited mount tree privatemount(abs_path, abs_path, NULL, MS_BIND | MS_REC, NULL)— bind the rootfs to itself so it's a mount pointmount("", abs_path, NULL, MS_PRIVATE | MS_REC, NULL)— make the rootfs subtree privatechdir(abs_path)→mkdir("old_root")→syscall(SYS_pivot_root, ".", "old_root")chdir("/")→umount2("/old_root", MNT_DETACH)→rmdir("/old_root")
Steps 1 and 3 exist because of two bugs fixed a day apart in Phase 2 (Errors #5 and #6), worth telling together: the first was silent in the way only mount propagation is silent, and the second was loud but grew from the same inherited propagation.
Error #5. The first version did steps 2 and 3 as one call: mount(abs_path, abs_path, NULL, MS_BIND | MS_PRIVATE, NULL). It returned 0. It was wrong. (That single-call version survives only in Error #5's write-up — the first committed setup_rootfs already has the split — so this one is anchored to the log, not to a tag.) mount(2) is explicit that the propagation-type flags — MS_SHARED, MS_PRIVATE, MS_SLAVE, MS_UNBINDABLE — cannot be combined with other operations in a single call, and rather than fail, the kernel silently dropped MS_PRIVATE and left the bind at the parent's default propagation, which on a systemd host is typically shared. Mount events inside the container could propagate out to the host. The runtime was reporting isolation it did not have. The fix is the two-call split you see above, plus MS_REC on both so submounts come along.
Error #6. The next day's fix was for pivot_root returning EINVAL on systemd systems. Systemd marks / as shared propagation at boot; CLONE_NEWNS copies that flag into the child's new mount table; and pivot_root refuses when the new root's parent mount is shared (man 2 pivot_root, EINVAL). Hence step 1: before touching anything, make the entire inherited tree private. This is safe precisely because we're already in a fresh mount namespace — the parent's view is untouched. runc, Docker, and LXC all do the same thing first.
Step 5's MNT_DETACH (Decision #12) is the lazy unmount: a plain umount returns EBUSY if anything still references the old root, whereas MNT_DETACH removes it from the namespace immediately and lets the kernel reap it as references drain. And — as Post 2 covered at length — the moment that detach happens, every absolute host path the child captured earlier stops resolving. The mount namespace is not a view you can casually cross back out of.
UTS: one flag, one call, in the right process
CLONE_NEWUTS (phase-4) is the simplest bit. It gives the child its own hostname and domain name, and the only work is sethostname(2). The one decision (Decision #17) is where to call it: in the child, after clone returns, never in the parent. CLONE_NEWUTS creates the new namespace for the child; the parent is still in the host's. Call sethostname from the parent and you have just renamed the host.
The alternative — unshare(CLONE_NEWUTS) from inside — was rejected for a reason that recurs throughout the project: clone is already being called with a flags word, adding a bit to it creates every namespace atomically in one syscall, and there's nothing to sequence. Whenever a namespace can be a clone flag rather than a separate unshare, it is.
IPC: the bit nobody notices until they need it
CLONE_NEWIPC (phase-4c) isolates System V IPC objects and POSIX message queues (man 7 ipc_namespaces). Inside the namespace the SysV tables start empty; a shared-memory segment or semaphore set created by a host process is simply not there. Phase 4c is a small phase — the flag, an --ipc option, and tests — and unlike the mount and PID bits, nothing implies it: --rootfs does not turn it on, only --ipc does (phase-4c:src/main.c). It's here because the isolation story is incomplete without it, and because a container that shares SysV IPC with the host is a quiet side channel.
User: the one that changes what "root" means
The first four namespaces need CAP_SYS_ADMIN to create — root, in practice. CLONE_NEWUSER (phase-4b) is the one that doesn't, and when it shares a clone call with the others the kernel creates the user namespace first and evaluates the rest against it (man 7 user_namespaces, "Interaction of user namespaces and other types of namespaces") — which is what makes minicontainer --user --pid /bin/sh work without sudo. It is also different in kind from the others: it doesn't just give the process a private view of some resource, it gives the process a private identity space and, within it, capabilities.
The mechanism is UID and GID maps. Inside the namespace the process can be UID 0. Outside, that UID 0 is your unprivileged UID — the kernel translates every credential check across the boundary. The map is written by the parent, into the child's /proc/<pid>/uid_map and /proc/<pid>/gid_map, and minicontainer's default is the simplest legal one (phase-4b:src/main.c:237-242, whitespace aligned):
.uid_map_inside = 0,
.uid_map_outside = getuid(),
.uid_map_range = 1,
.gid_map_inside = 0,
.gid_map_outside = getgid(),
.gid_map_range = 1
One line each: "UID 0 in there is my UID out here, and that's the only mapping." An unprivileged parent is only permitted to map its own effective UID and GID, in a single-line map (man 7 user_namespaces, "Defining user and group ID mappings") — anything more goes through the newuidmap/newgidmap helpers and a /etc/subuid allocation (man 5 subuid), which minicontainer does not use.
There is a required ordering in setup_user_namespace_mapping (phase-4b:src/uts.c), the kind of thing the man page buries in a sub-bullet: for an unprivileged writer, /proc/<pid>/setgroups must be written "deny" before gid_map can be written at all. The kernel enforces this because a process that could both keep setgroups(2) and remap its GIDs could drop supplementary groups that were restricting it. So the one kernel-imposed order is setgroups ← "deny" before gid_map; minicontainer writes setgroups, then uid_map, then gid_map, each a one-line write.
Which raises a sequencing problem: the parent writes the map, but the child is already running. Until the map lands, the child has no identity of its own: its getuid() returns the overflow UID, 65534 by default (man 7 user_namespaces, "Unmapped user and group IDs"). minicontainer's rule is simple: the child does no work at all until the maps are written. That is the sync pipe: the parent creates a pipe, hands the read end to the child, the child's first act after clone is a blocking one-byte read, and the parent writes "1" only after the three /proc writes succeed (phase-4b:src/uts.c::child_func — "blocks on the sync pipe until the parent has finished writing UID/GID maps"). The primitive turned out to be general: Phase 6 widened the same pipe to gate network setup, and the Phase 7b fix for Error #25 (Post 2) widened it again so the child couldn't run before it had been placed in its cgroup.
Two smaller decisions round out the phase. The user namespace was folded into uts.c rather than given its own user.c (Decision #18) because, unlike UTS or overlay, it adds no new child-side function — the child just blocks on a read; all the work is parent-side. And the mapping helper originally took the whole uts_config_t, so the Phase 5 and 6 callers took the cheaper route of building a synthetic one just to pass seven fields through; Phase 7a replaced that with a dedicated user_ns_mapping_t holding exactly those seven — six map fields plus enable_debug (Decision #31) — which is why the current signature looks the way it does.
The first veto: /proc under a user namespace
The user namespace is also where the host's security policy first shows up as a participant rather than a bystander.
Mounting /proc inside an unprivileged user namespace has a kernel-side rule: the new mount may not be less restrictive than the proc already mounted on the host. Most distributions mount it MS_NOSUID | MS_NODEV | MS_NOEXEC, so Phase 2's mount("proc", "/proc", "proc", 0, NULL) — flags zero — is rejected with EPERM inside a user namespace. Decision #19 changed the flags to match in Phase 4b, which is also simply correct hardening (nobody needs setuid binaries or device nodes under /proc, and Docker, Podman, and runc all mount it this way).
But even with correct flags, the mount can still be denied: on Ubuntu 23.10 and later, the kernel or AppArmor may refuse proc mounts from inside an unprivileged user namespace. This is a platform restriction, not a bug in the runtime. So Decision #20 makes the failure graceful when a user namespace is active — a warning and continue, instead of an abort (phase-4b:src/uts.c:237-246, reindented):
if (mount_proc(args->enable_debug) < 0) {
if (args->user_namespace_active) {
fprintf(stderr,
"[child] Warning: /proc mount denied (user namespace restriction) "
"— continuing without /proc\n");
} else {
fprintf(stderr, "[child] Failed to mount /proc\n");
return 1;
}
}
Two things make this safe rather than sloppy. First, most commands don't need /proc — ls, echo, hostname, id, a shell all run fine; only ps-shaped tools care. Second, the fd-cleanup step that protects against the CVE-2024-21626 class normally iterates /proc/self/fd, and it has a brute-force fallback (close every fd from 3 up to RLIMIT_NOFILE) that engages automatically when /proc isn't there — so degrading /proc does not open a security gap. Without --user, a failed /proc mount stays fatal, because in the privileged path it means something is genuinely misconfigured.
That was the first place the host could overrule a step the runtime had configured correctly. The second was the one that taught me what "rootless" means.
The wall: a sethostname the host would not allow
The documented rootless example — minicontainer --user --pid --hostname mycontainer /bin/sh -c 'id && hostname' — failed on my Ubuntu 24.04 machine with:
sethostname: Operation not permitted
[child] Failed to setup UTS
Same binary, same flags, same user: it succeeded when run from the VS Code integrated terminal and failed from a plain gnome-terminal, an SSH session, or a tmux pane. Nothing in the runtime differed between the two. This is Error #19, and it is not a code bug — it is the host exercising a veto.
The root cause is a sysctl Ubuntu 24.04 ships enabled: kernel.apparmor_restrict_unprivileged_userns=1. With it set, when an unconfined process calls clone(CLONE_NEWUSER), the kernel auto-attaches a restrictive default AppArmor profile — unprivileged_userns — to the new namespace. That profile masks CAP_SYS_ADMIN inside the namespace. So the child is "root" in its own user namespace, the CLONE_NEWUTS bit was set correctly, the UID map was written correctly, the sync pipe fired in the right order — and sethostname(2), which requires CAP_SYS_ADMIN in the UTS namespace's owning user namespace, returns EPERM anyway. From inside the process there is no way to see why. The namespace was "successfully created." The capability is simply gone.
Why did VS Code's terminal work? Because it isn't unconfined. It runs under the snap.code.code AppArmor profile in complain mode — cat /proc/self/attr/current prints snap.code.code (complain) there, versus unconfined from the gnome-terminal shell. A complain-mode profile logs violations but doesn't enforce them, and the auto-attach mediation only fires for processes the kernel treats as unconfined at the moment of namespace creation. The one place the rootless example worked was, in effect, an accident of which sandbox the parent shell happened to be sitting in.
It reaches further than the quickstart. The unit-test suite runs the same rootless hostname case unprivileged: make test invokes ./test_uts once without sudo (the Makefile step labeled "runs unprivileged subset too"), and tests/test_uts.c::test_user_namespace_with_hostname builds a config with the user namespace, the UTS namespace, and a hostname, then asserts the container exited 0 — Error #19's repro, as a unit test. The shipped profile doesn't cover it, either: make install-apparmor renders the minicontainer binary's path into the profile, so test_uts is a separate, unconfined executable. By construction, then, on a stock 24.04 box that one test case tracks the AppArmor confinement of whatever shell launched make.
That is as far as the profile's reach goes, and an earlier version of this paragraph took it further than the evidence allows. My runs were green from the VS Code terminal and failed under headless automation, and for months I had filed that as "the suite needs a TTY." When I found the profile, I rewrote the story: the TTY was a correlate, the profile was the cause. It isn't, and it wasn't. Give make test a stdin that is not a terminal and it dies at its first step — sudo ./test_core, as root, four steps before the unprivileged test_uts ever runs — on an assertion AppArmor has nothing to do with. Since phase-7b, container_config_t has carried stdout_fd and stderr_fd fields whose documented sentinel is -1 for "inherit" (phase-7b:include/core.h:80-81). The CLI sets them (phase-7b:src/cli.c:268-269). The tests don't: every test file builds its config with a designated initializer that never names them (phase-7b:tests/test_core.c:13-23), so both are 0, and the child's guard — if (args->stdout_fd >= 0) dup2(args->stdout_fd, STDOUT_FILENO) (phase-7b:src/core.c:206-207) — is true for 0. The container's stdout and stderr become whatever stdin was. From a terminal that is a read-write pty, and echo $$ lands on your screen looking like success. From a pipe or < /dev/null it is read-only; write(2) fails with EBADF, the shell exits non-zero, and the assertion fires. Two commands, differing by one character, as root, at any tag from phase-7b on: ./test_core < /dev/null aborts; ./test_core 0<> /dev/null passes. Five of the thirteen suites at phase-8c fail this way — every one whose container command writes to stdout. The AppArmor mediation touches one test case, and you can only see it once stdin is writable.
So the record is: two independent environment dependencies, and the one shell the suite passed from happened to satisfy both — a read-write pty and a complain-mode profile. The TTY was not a correlate. It was the larger of the two causes, and the correction that replaced it with the profile was itself the kind of mistake this series is about: a mechanism found, and declared the whole story. At phase-8c, the current tag, both halves are still as described (src/core.c:298-299, tests/test_core.c:13-23); the fd bug is not yet fixed in any tagged release.
Three fixes were considered, and the log records the reasoning. Detect the mediation and skip sethostname with a warning — rejected, because silently ignoring a flag the user passed is worse than surfacing the kernel's verdict. Ship a per-binary AppArmor profile declaring userns, — rejected in the error entry as "too invasive for an educational project" — an install step that "pulls the project out of 'drop-in, no-config' territory" — to be reconsidered in Phase 8b alongside the hardening work. Document it — chosen: the mediation is the system doing exactly what the sysctl asks, raising the unprivileged-userns baseline is a reasonable thing for a distribution to do, and the right response is to teach where the boundary is.
And then the profile shipped anyway — in the same commit. scripts/apparmor/minicontainer.profile.in landed in the commit that shipped Phase 7a (5f06b30, 2026-05-14), and make install-apparmor renders the binary's path into it and loads it as /etc/apparmor.d/minicontainer. It contains exactly one rule — userns, — under flags=(unconfined), plus an include if exists <local/minicontainer> hook for site additions; it is not a sandbox, just the same narrow pattern Docker, Podman, Chromium, and Firefox use for their own unprivileged-userns needs. The README now presents that one command as the recommended fix, with the alternatives being a complain-mode-confined parent, sudo (and giving up rootless), or disabling the sysctl host-wide, which it marks not recommended because it lowers the whole system's baseline. One footnote for anyone checking my work: Error #19's original analysis still reads "Rejected (for now)" — and the entry was written in the very commit that shipped the profile, so it contradicted the repository from day one; nobody noticed until this post. It now carries a dated update (2026-10-12) recording what actually shipped, so the entry and the repository agree again.
The principle: namespaces compose, but the host holds a veto
Line the five up and the pattern holds cleanly for four of them. PID, mount, UTS, IPC — each is a bit, each is independent, each does what the flag says. The bugs in those phases (#5, #6) were about mount-propagation mechanics — which flags may share one mount() call, and what the inherited tree looks like — and once those were right the kernel delivered exactly the isolation promised.
The user namespace is the exception. The other four isolate a resource: a process table, a mount table, a hostname, an IPC table. The user namespace manufactures authority — a private root — and unprivileged authority is exactly what a distribution's security policy is tuned to arbitrate. In this build, run as root from an unconfined shell, the LSM never showed up for the first four bits; it showed up the moment the fifth let an unprivileged user ask for the same things. The kernel or AppArmor may deny the /proc mount (Decision #20). AppArmor may mask CAP_SYS_ADMIN and deny sethostname (Error #19). Error #19's lesson generalizes this to SELinux, Landlock, and lockdown mode — I have not reproduced those — but the shape it describes is the same: a knob that produces EPERM from inside a namespace that was, by every measure the program can take, created successfully.
Which is the lesson Error #19 draws, and the one I'd want an infrastructure engineer to leave with: "rootless" is a property of the runtime, not of the host. The runtime can guarantee it set the right flags in the right order. It cannot guarantee the host will let those flags mean what they usually mean. And because the user's first encounter with a rootless feature is the one that forms their impression, the documentation has to cover the LSM interaction before they hit it — "rootless doesn't work on my distro" sticks; "rootless works once you allow userns, for this binary" only forms if someone told you.
Check my work
The CLI invariants and the default mapping are at phase-4b:src/main.c (lines 228–230 and 237–242); the flag assembly is phase-4c:src/uts.c::uts_exec (lines 349–381, including the CLONE_NEWIPC bit). The five-step pivot is phase-2:src/mount.c::setup_rootfs, with Decisions #10 (why not chroot), #11 (the order and the two-call rule), #12 (MNT_DETACH), and Errors #5 and #6 in docs/decisions.md. UTS is Decision #17 at phase-4. The user-namespace mapping sequence — setgroups, uid_map, gid_map — is phase-4b:src/uts.c::setup_user_namespace_mapping, with the sync write that follows it in uts_exec, plus Decisions #18 and #31 and the /proc rules in Decisions #19 and #20. The AppArmor wall is Error #19 in full, including the /proc/self/attr/current outputs and the four workarounds, and the same material is in the README under "sethostname: Operation not permitted." External authorities: man 7 pid_namespaces, man 7 user_namespaces, man 7 ipc_namespaces, man 2 pivot_root, man 2 mount, man 2 sethostname. If you're on Ubuntu 24.04, the repro is one command from an unconfined shell.
Next in the series: cgroups v2 — wiring memory, CPU, and PID limits by hand, and the placement-ordering bug that left the memory and PID caps unenforced until someone thought to watch memory.current.
Comments (0)
No comments yet. Be the first to comment.
Leave a Comment