fds Are Namespace-Agnostic, Paths Are Not

The first post in this series made a promise: every claim would come with a way to check it — a git tag, a docs/decisions.md error number, a command you can paste. This is the post that promise was written for. It is a tour of the places where the Linux kernel refused to honor an assumption I would have sworn was safe, drawn from the numbered error log of minicontainer, the educational container runtime I built one isolation boundary at a time.

Each story has the same shape: the assumption, the symptom, the root cause, and the one-line lesson. Every one of them is a documented error in that log — except the first, which I caught while designing, before it could ship, and which is the cleanest statement of the rule the others broke. And there is a second thread running underneath, which I did not notice until I lined the bugs up next to each other: almost none of them were caught by the test suite, and the reason they weren't is the same reason every time.

I'll get to that. Start with the rule.

fds are namespace-agnostic, paths are not

To run an interactive shell, the container needs a pseudo-terminal. The obvious design — the one I sketched first — is to allocate the pty in the parent with posix_openpt, get the slave's path from ptsname (something like /dev/pts/3), and pass that string across clone to the child, which opens it and wires it up as its controlling terminal. Clean. A path is a path.

Except a path is not a path. The container mounts its own devpts instance — a fresh, isolated /dev/pts (mount_devpts, unconditional once a rootfs is set, because apt and nano need a working pty subsystem). The /dev/pts/3 that ptsname handed back in the parent was minted by the host's devpts. Inside the child's newinstance devpts, /dev/pts/3 either doesn't exist or names an entirely different terminal. The string survives the clone intact and means nothing at the destination.

The fix inverts the whole flow. Open the pty pair inside the child, after mount_devpts, so it comes from the container's own devpts — and then send the master file descriptor back out to the parent over a SCM_RIGHTS control message on a pre-clone socketpair. At phase-7b, that's pty_open_in_child followed by pty_send_master; include/pty.h describes the master end in one line: "Child sends to parent via SCM_RIGHTS."

Why does the fd cross the boundary when the path can't? Because they are different kinds of thing. A path is a name resolved against a mount namespace; change the namespace and the name resolves differently, or not at all. A file descriptor is a handle to a kernel object — an entry in the process's file table pointing at an open file description the kernel owns. SCM_RIGHTS doesn't copy the number; it installs a new reference to the same underlying object in the receiver's table. The master doesn't care which devpts instance is mounted where, or that the parent can't see the container's /dev/pts at all. It's a reference, not a name.

Lesson. When you need to move a terminal, a socket, or any open resource across a namespace boundary, move the descriptor, not the path. Paths are namespace-relative; descriptors are not. I saw this one coming, which is the only reason it's here — it never shipped. The next one is the same law with teeth.

The same law, with teeth: a bind mount that mounted the wrong thing

--volume /tmp/data:/mnt should bind-mount a host directory into the container. The first implementation applied the volumes in a loop in the child, right after setup_rootfs, resolving each source with stat on the absolute host path captured at parse time. It failed for every volume whose source wasn't also present inside the rootfs, and the --debug trace said exactly why:

[child] pivot_root successful
[child] Unmounted old root
[bind] host_path /tmp/voltest: No such file or directory
[child] Failed to apply bind mount /tmp/voltest -> /mnt
[parent] Child exited: 1

setup_rootfs does the full pivot_root dance and then umount2("/old_root", MNT_DETACH). That detaches the host root from the child's mount namespace. By the time the bind loop runs, the absolute host source — /tmp/voltest, perfectly real on the host — no longer resolves in the namespace the child now lives in. stat returns ENOENT, the child aborts before execve. This is Error #22, and it is precisely the mount-side twin of the pty story: an absolute path string is only meaningful in the mount namespace that was current when it was produced, and pivot_root + MNT_DETACH is a namespace transition that silently invalidates every host path captured before it.

The cruel part is how it hid. If the source path also exists inside the rootfs — --volume /tmp:/mnt, and the rootfs ships its own /tmp — the post-pivot stat succeeds against the rootfs's inode, mount returns 0, and nothing errors. It has just bind-mounted the container's empty /tmp onto the target instead of the host's. Wrong source, silently, with a zero exit code. Every manual test that reached for a familiar path — /tmp, /etc, /home — landed on one present in both trees and read as "worked, directory was empty." It took a host-only scratch path (/tmp/voltest, created on the host, absent from the rootfs) to make the ENOENT deterministic.

The fix: apply the binds inside setup_rootfs, before pivot_root, while the host source still resolves, composing the target under the future root so the pivot carries it along.

Lesson. Establish the bind while the source still resolves — before the old root is severed, not after. A path captured on one side of a namespace transition is a dangling reference on the other side, and a dangling reference that happens to hit something is worse than one that errors. (Error #22, phase-7b.)

setns won't let you into a room you're already in

minicontainer exec joins a running container's namespaces the standard way: iterate the namespace types, open each /proc/<pid>/ns/<type>, and setns into it. The original loop treated a missing file as "the container didn't create this namespace, skip it," and any other failure as fatal:

int fd = open(path, O_RDONLY | O_CLOEXEC);
if (fd < 0) { if (errno == ENOENT) continue; ... return 1; }
if (setns(fd, NS[i].flag) < 0) { /* fatal */ return 1; }

user was first in the list. And exec aborted with setns(user): Invalid argument for every ordinary container — the common case, on every host.

The assumption baked into the ENOENT skip is that /proc/<pid>/ns/user is absent when the container didn't make its own user namespace. It is never absent. Every process belongs to some namespace of every kind, including the initial ones, and the /proc entry always exists. So file-presence cannot tell "the container created a distinct user namespace" from "the container shares mine." A container started without --user shares the init user namespace with the exec caller — and the kernel's userns_install rejects joining a user namespace you are already a member of. setns(fd, CLONE_NEWUSER) returns EINVAL when the target is your current user namespace. A fifteen-line repro confirms it: setns on your own /proc/self/ns/user gives errno 22.

The fix keys on the right signal — namespace identity, not file existence. Before joining, fstat the target and stat your own /proc/self/ns/<type>, and if the st_dev/st_ino pair matches, skip it; you're already there. Comparing those inodes is the canonical way to ask whether two nsfs references name the same namespace.

Lesson. "Did this container get its own namespace of type T?" is an identity question — compare the inodes — not an existence question. The ENOENT-only skip answered the wrong question and worked only by luck, on the one configuration (--user, which does have a distinct user ns to join) that was plausible to test with. (Error #23, phase-7b.)

The limit that had the right value and stopped nothing

--memory 64M, --pids 15. The cgroup gets created, memory.max and pids.max get the right numbers written into them, the container runs. And then a fifty-process fork loop runs to completion, and a 200 MB allocation succeeds, under a 15-process, 64 MB cap.

The limits weren't wrong. They weren't applied. The two-step cgroup pattern is: create the cgroup and write the limits in the parent, clone the child, then write the child's PID into cgroup.procs to move it in. The bug was in the word then. The parent placed the child into the cgroup after signaling it to proceed — and for a plain --memory/--pids container with neither --user nor --net, there was no signal at all, because the sync pipe was only created for those two features:

bool needs_sync = enable_user_namespace || enable_network;   /* NOT cgroup */
...
/* Step 11 */ if (needs_sync) write(sync_pipe[1], "1", 1);   /* unblock child */
/* Step 12 */ if (enable_cgroup) add_pid_to_cgroup(pid);     /* too late */

So the child ran the instant clone returned — forking, faulting memory — while still a member of the parent's cgroup. /proc/<child>/cgroup read from inside proved it: the container's process was sitting in the launching shell's scope, …/app-…Chromium….scope, not minicontainer_<id>. The placement did eventually happen, after the workload had already raced ahead, which is why the occasional late fork did get denied and the whole thing looked intermittent. And memory.current on the container's cgroup read 0 for a running container, because cgroup v2 charges pages to whichever cgroup the process was in when it touched them and does not re-charge on migration.

This one is Error #25, and it has the longest tail in the log: it dates to Phase 5, where the pattern was introduced with a documented rationale — the error log's post-mortem quotes it — that it "closes the window … where the child is running without limits." That claim is false unless the child is blocked until placement is done. It stayed latent through Phases 5, 6, 7a, and into 7b, copied forward byte-for-byte by a refactor that was careful to preserve behavior, including this one. The fix is two lines of ordering: widen the sync gate to include enable_cgroup so a pipe always exists, and move the placement to before the unblock. The child doesn't run until it's a member.

Lesson. "The limit file has the right value" and "the limit is enforced" are different claims, and only the second one requires the placement to win a race against the workload. The deterministic way to win the race is to gate the child until placement is done — creating the cgroup and writing a PID into cgroup.procs is not enough if the process has already started doing the thing you meant to constrain. (Error #25; origin phase-5, fixed phase-7b.)

The flag that rejected itself

minicontainer run --secure --pid --rootfs ./rootfs … exited with Error: --secure requires --rootfs. With --rootfs right there on the command line. Drop --secure and the identical command ran fine, so --rootfs parsing was not the problem. --secure was simply inert: it rejected every invocation it was asked to guard.

The validation looked correct:

if (out_cfg->enable_hardening && !out_cfg->rootfs_path) { ...reject... }

parse_run_flags collects options into local variables during the getopt loop and only commits them to the output struct in a config-build block about eighteen lines below this check. At the point the guard runs, out_cfg->rootfs_path is still NULL — the caller zero-initialized the struct — so !out_cfg->rootfs_path is always true whenever --secure is set. The flag rejected itself from the first build. The --overlay guard sitting a few lines above it had been right the whole time, because it tested the local rootfs_path, the variable that actually holds the parsed value at that point in the function.

Lesson. A validation that reads its destination field is fragile when that field gets populated later in the same function. Check the source local, not the not-yet-written output. (Error #26, phase-8b.)

A flat list is a lossy way to say (flags, data)

An OCI bundle's config.json can ask for a tmpfs with options like ["nosuid","nodev"]. Running such a bundle aborted the child instantly:

mount(tmpfs): Invalid argument
[child] Failed to apply OCI mount tmpfs -> /mnt-tmp

A tmpfs with no options worked. A tmpfs with only mode=755 worked. Since real umoci/buildah bundles routinely mount tmpfs at /dev with ["nosuid","strictatime","mode=755","size=65536k"], this broke essentially every real-world bundle. The consume side joined the options array into a comma string and passed it to mount(2) as the fifth argument:

mount("tmpfs", m->destination, "tmpfs", 0, m->options);  /* "nosuid,nodev" as data */

But the OCI options array conflates two different things. nosuid, nodev, ro, noexec are VFS mount flags — they belong in the fourth argument, mountflags, as MS_NOSUID, MS_NODEV, and friends. mode=, size= are filesystem-specific data and belong in the fifth. tmpfs's data parser has never heard of nosuid as a data key, so it rejects the whole mount with EINVAL. The runtime is responsible for splitting the array the way mount(8) does — mapping the known flag tokens to their MS_* bits and passing only the leftovers as data.

Lesson. A flat array of option strings is a lossy encoding of a (flags, data) pair, and the consumer has to restore the split. The naive mount(…, 0, options) compiles, parses, and passes every parser-level test while being wrong against any real input that carries a flag token. (Error #27, phase-8c.)

Lightning round: three that hid in plain sight

sudo lied to my test. Early on, testing the container via sudo showed no leaked file descriptors inside it — ls /proc/self/fd returned only 0, 1, 2 — which made the absence of any fd-cleanup code look perfectly safe. It wasn't safe; sudo was hiding it. sudo spawns a fresh process with a minimal fd table and does not forward the arbitrary descriptors open in your shell (editor buffers, redirections, multiplexer sockets). The leak the test was supposed to catch never reached the runtime, because the test harness itself stripped it. The real exposure — the CVE-2024-21626 class — is fds opened by the runtime's own code, which sudo does nothing about. (Error #12.)

strstr found the wrong pid. The hand-rolled state-file parser looks up a key by searching for the quoted string and taking the first hit. state.json had a top-level "pid": 12345 (the integer PID) and, inside a "namespaces" object, a "pid": true (the boolean). extract_bool(buf, "pid", …) found the integer first, failed to parse 12345 as true/false, and silently left the namespace flag at its false default. This is the one bug in this whole post that got caught early — because Phase 7b added a state_save→state_load round-trip unit test, and it failed on the first run with pid_ns mismatch. Hold that thought. (Error #20.)

The kernel dropped a flag I passed it. mount(src, dst, NULL, MS_BIND | MS_PRIVATE, NULL) — one call to bind a directory and mark its propagation private. The kernel silently ignores MS_PRIVATE when it's combined with MS_BIND, leaving the mount at the parent's default propagation (usually shared), so mount events leak across the very namespace boundary you built. Propagation changes have to be their own mount() call. On stricter kernels the same shared-propagation root can leave pivot_root itself failing with EINVAL. (Error #5.)

The pattern: everything broke one layer below the test

Line these up and the second thread is impossible to miss. The bind mount hid because the unit test exercised the bind helper inside a plain unshare(CLONE_NEWNS) child with no pivot_root, so the host source was always reachable — the seam between the pivot and the helper was covered by neither the unit nor the integration suite. setns-into-your-own-userns hid because no test reaches exec at all; it only means anything against a live, externally-started container, and the suite builds container_config_t in memory and calls the core directly. The silent cgroup hid because the tests asserted the shell exited normally and that memory.max held the right value — never that a fork was actually denied. --secure hid because every test builds the config struct by hand and never once calls parse_run_flags, so the entire CLI-validation layer was untested. The tmpfs EINVAL hid because the OCI tests validate the parser and translator — that the options join to "nosuid,nodev" — and never call mount(2), which needs root and a real mount namespace.

Same shape every time. The code built a configuration in memory and asserted a value. The bug lived in the part the test never reached: the syscall it never made, the CLI parse it bypassed, the fd table sudo never populated, the second half of a namespace transition. A container runtime is mostly a careful sequence of privileged syscalls made in a cloned child, and a test that constructs a struct and checks a field is testing the easy half — the half that was already going to be right.

Which is exactly why the strstr bug is the tell. It was the one that got caught before it could embarrass me in production, and it got caught for one reason: someone finally wrote a test that saved a real file and read it back — a test that crossed the seam. The lesson of this whole post is in that contrast. The kernel's contracts are simple, and they have corners; you find the corners by reproducing failures against the real thing, as root, in a real namespace — not by asserting that a limit file contains the number you just wrote to it.

Check my work

Every numbered story here lives in docs/decisions.md in the repo, under the error number in parentheses, each with its full symptom, root cause, fix, and the date the fix landed. The fixed code lives at the tag named alongside it — git checkout phase-7b for the pty, the bind mount, setns, and the strstr key collision; phase-8b for --secure; phase-8c for the tmpfs split; and the cgroup bug's origin at phase-5, its fix at phase-7b; the two oldest — the dropped MS_PRIVATE and the sudo-masked fd leak — go back to phase-2, with the fd cleanup that fixed the latter landing at phase-3.

And if you want the pre-fix code rather than the log's quotation of it, it's checkable to different depths. The broken cgroup ordering is genuinely sitting in the tree at phase-5 through phase-7a, exactly as described. The bind-mount and setns bugs are in the commit history between phase-7a and phase-7b. The --secure and tmpfs bugs never reached a commit at all — both were caught on live runs before their phases shipped, so the error log's verbatim quotes are what remains of them; though the fixed --secure guard at phase-8b still carries a comment explaining exactly the mistake it refuses to repeat. Don't take my word for any of it.

Next in the series: the namespaces themselves — what each one actually buys you, why pivot_root beats chroot, and the afternoon an AppArmor profile refused a rootless sethostname the kernel had every reason to allow.