The Same Producer-Consumer Buffer In Three Runtimes, And What The Comparison Reveals

Two of these codebases are halves of the same data pipeline. The third is from a different project entirely. All three face the same producer-consumer shape — many producers writing into a shared place, one consumer draining it — and the shape is universal enough that one project's pipeline alone has it twice, in two completely different runtimes on opposite sides of an HTTPS boundary. An ESP32 gateway firmware takes ESP-NOW packets from sensor nodes in through a FreeRTOS critical section, draining them every sixty seconds into a cloud POST. The packets become a JSON POST that lands at a Go HTTP service on a VPS, where goroutines fan in to a Postgres connection pool and a Prometheus registry — a different kind of buffered fan-in with the same shared-state problem at its heart. The third instance, a Python TCP server I wrote for an unrelated CTF lab, rounds the comparison out: per-connection threads append each request to a shared append-only file, and the absence of any explicit lock is correct anyway — for a reason split between CPython's buffering and the kernel's append, not the GIL you would reach for first.

Three runtimes, three answers. The ESP32 firmware wraps its buffer in a portENTER_CRITICAL macro whose mechanism changes with the number of cores on the chip — a real cross-core spinlock on the dual-core part it runs on, and nothing but an interrupt-level raise on the single-core chip the project used to run. The Python server has no user-authored synchronization primitive, and is correct anyway — for a reason that turns out to be split between CPython's I/O buffering and the operating system, not the GIL you would reach for first. The Go service has no sync.Mutex in the user code either, for entirely different reasons, and is race-safe at the shared-state level because of a discipline that the language’s library ecosystem makes the path of least resistance. The solutions are as different as solutions to the same problem can be, and the fact that two of them sit on opposite sides of an HTTPS boundary in a single project — and still need different answers — is the more interesting half of the observation.

This post is a side-by-side reading of the three approaches — and then the inverse case, where the consumer fans out instead of the producer. The comparison is not a survey of concurrency primitives — there are several excellent surveys, and I will reference them rather than try to re-write them — but a specific argument about a thing the surveys do not always foreground: that the right primitive is primarily a function of the runtime's sleep model and of which side of the buffer fans out, and that picking the wrong primitive produces a system that works in testing and fails under load in ways that look mysterious from the outside.

The post is long. I will try to make it worth the length by walking the comparison carefully and being specific about each primitive's mechanism rather than its name.

The shared-state problem, sketched

At the level that matters for this comparison, each codebase poses the same shape: independently-driven writers converge on shared state, and something has to keep their writes from colliding. Only the ESP32 case is a literal bounded in-memory buffer drained by one consumer; the Python and Go cases expose the same coordination question through different storage — a shared append-only file, a shared connection pool and metric registry. The classical producer-consumer questions still apply: how does a reader observe the shared state without missing items a writer is concurrently adding? How does a writer handle state that is currently being drained? How does the system avoid the lost-update problem in which two writers' updates collide and one is lost?

In each of the three runtimes, the answer to all three questions falls out of the runtime's sleep model — the rules that govern when execution can be suspended and resumed and what guarantees the runtime makes about visibility and ordering across those suspensions. The sleep model is the load-bearing concept of this post and the thing each of the three sections will return to. I will define it gradually rather than abstractly, which is the way I came to understand it.

Answer one: portENTER_CRITICAL on a dual-core Xtensa ESP32

The gateway firmware runs on the classic ESP32 — a dual-core 240 MHz Xtensa LX6 part, the esp32dev target — under FreeRTOS. Keep its single-core predecessor in mind as we go: an earlier single-probe gateway ran the same family of code on a single-core RISC-V ESP32-C3 (since retired), and the same source line will turn out to mean something different on each. The code that buffers inbound ESP-NOW packets is roughly the following, simplified:

typedef struct {
    char node_id[16];
    float temp_c;
    /* ...other fields... */
} reading_t;

reading_t buffer[MAX_BUFFER_SIZE];
volatile int buffer_count = 0;
portMUX_TYPE buffer_mux = portMUX_INITIALIZER_UNLOCKED;

void on_data_recv(const uint8_t *mac, const uint8_t *data, int len) {
    /* called from the ESP-NOW receive callback, which runs in ESP-IDF's WiFi task at higher priority than the main loop */
    if (len < (int)sizeof(reading_t)) return;   /* ignore short packets */
    portENTER_CRITICAL(&buffer_mux);
    if (buffer_count < MAX_BUFFER_SIZE) {
        memcpy(&buffer[buffer_count++], data, sizeof(reading_t));
    }
    portEXIT_CRITICAL(&buffer_mux);
}

void post_readings(void) {
    /* called from the main loop, every 60 seconds */
    reading_t snapshot[MAX_BUFFER_SIZE];
    int count;
    portENTER_CRITICAL(&buffer_mux);
    count = buffer_count;
    memcpy(snapshot, buffer, sizeof(reading_t) * count);
    buffer_count = 0;
    portEXIT_CRITICAL(&buffer_mux);
    /* ...send snapshot to the cloud over HTTPS... */
}

The thing called a portMUX_TYPE is a spinlock, and on this dual-core part portENTER_CRITICAL genuinely acquires it. But the spin only arbitrates between the two cores — the calling core never spins against itself; a re-entrant take by the owner is tracked as recursion, not a deadlock. Same-core exclusion comes from the other half of the macro: it raises the interrupt level high enough to lock out the scheduler and every ordinary interrupt. One macro, two mechanisms running at once, and which of them is load-bearing depends on which core you are asking about.

This is the part worth slowing down on. portENTER_CRITICAL names a contract — “no participating path can enter the region guarded by this mux during the body” — without committing to a single mechanism, and the mechanism it emits differs between a dual-core chip and a single-core one.

If you trace portENTER_CRITICAL through the ESP-IDF source (the Xtensa and RISC-V FreeRTOS ports), the dual-core target does both halves. It takes the portMUX_TYPE's spinlock so the other core cannot enter a critical section guarded by the same mux, and on the calling core it raises the interrupt level — on the Xtensa ESP32 by writing PS.INTLEVEL up to the exception level (XCHAL_EXCM_LEVEL, which is 3 on this part), masking every ordinary interrupt and, with it, the timer interrupt that drives the scheduler. Genuinely high-priority level-4/5 and NMI sources stay live, but none of them touch the buffer. The spinlock is the cross-core half; the interrupt-level raise is the same-core half. The single-core RISC-V predecessor had only the second half: there the spinlock compiled out to a no-op — no other core to contend with — and mutual exclusion came entirely from raising the CPU's interrupt-threshold register to the same exception level, not from blanket-clearing the global interrupt-enable bit.

What the critical section guarantees is that, during its body, no other path that takes the same mux can touch the buffer. No ordinary interrupt service routine runs on this core, because they are masked. No other FreeRTOS task is scheduled in, because the scheduler is itself driven by a timer interrupt and that interrupt is masked. And no code on the other core can enter the matching critical section, because the spinlock is held. It is a full mutual-exclusion barrier against everything that could race the buffer, on either core.

This is the right primitive for the gateway firmware's buffer for two related reasons. The ESP-NOW receive callback runs from ESP-IDF's WiFi task — a separate FreeRTOS task from the main loop's app task, documented as one whose callbacks should do minimal work and hand data off rather than process it inline. The buffer needs some form of mutual exclusion, since the WiFi task and the app task can be scheduled in either order and, on this dual-core part, can genuinely run at the same instant on different cores. A FreeRTOS mutex would technically satisfy the contract, since both contexts are tasks that can sleep on a wait. But the critical section is short — a memcpy of a few dozen bytes plus an integer increment — and a mutex can suspend the task and carries the RTOS's wait-and-priority bookkeeping, while portENTER_CRITICAL never sleeps — it costs only the interrupt-level raise plus, on the dual-core part, a brief spin if the other core is holding the same mux. For a critical section measured in microseconds and a producer that runs to completion within those microseconds, this is the right shape.

The cost of the design is that ordinary interrupts are masked, on the running core, for the duration of the critical section. During those microseconds, interrupt servicing and task scheduling on that core are deferred — anything the WiFi driver wanted to dispatch into a callback waits for the critical section to exit. For a sensor cadence of one packet per minute per node and a critical section of a few microseconds, the deferral is invisible. For a system in which the producer fires at a much higher rate, or the critical section did meaningful work, the cost would become measurable: ESP-IDF specifically warns that pending interrupts are not serviced during a critical section, and that critical sections should contain only a few data-structure or hardware-register accesses for that reason.

The portMUX_TYPE itself, the bookkeeping struct the macros take by reference, is real on this chip: it holds the spinlock value the two cores contend over, plus an owner-core identifier and a recursion count. On the single-core predecessor that struct was vestigial — the spinlock value was never contested and the owner field was debug-only — which is the giveaway that the single-core build was a degenerate case of the dual-core one, with the spinning compiled away. The same source declares the same struct; only one of the two chips actually uses its contents.

The point to land before moving on is that the contract of portENTER_CRITICAL — "nothing else enters a region guarded by the same mux while the body runs" — is the same on single-core and dual-core chips, but the mechanism is different. The contract is what the user of the API can depend on. The mechanism is what the implementation does to satisfy the contract. The contract is portable; the mechanism is not. APIs that name contracts rather than mechanisms are the ones that survive being ported between architectures, and the ones whose names you should not over-interpret.

That is the contract-versus-mechanism distinction made concrete by a real chip change. The project's gateway used to run on the single-core C3; it now runs on the dual-core ESP32, and the C3 is retired. The buffer's synchronization did not change in kind across that move — it was portENTER_CRITICAL over a portMUX_TYPE before and after, providing the same contract: no other path that takes the same mux touches the buffer during the body. What changed was the machine code underneath. On the C3 the macro was an interrupt-level raise and nothing else; on the dual-core part it is that same raise plus a real cross-core spinlock, and the portMUX_TYPE that was vestigial bookkeeping on the C3 became live state the moment there was a second core to contend with. The same primitive, the same contract, different instructions on different chips — and the only thing the source had to know across the change was the contract.

The gateway firmware also gained a second producer-consumer pairing on top of the buffer above, for a different reason. The original on_data_recv callback in the example does roughly the producer's work inline — memcpy, validate, allocate a slot, write fields. That worked at one packet per minute per sensor. It stopped working when the firmware started receiving a burst of packets per wake from a multi-probe sensor — a state packet plus one reading per probe, a millisecond or two apart. The WiFi driver's small RX queue could not absorb the burst while the callback was busy with per-packet work, and the second packet in each burst started getting dropped silently. The fix was a second producer-consumer pairing between the WiFi callback and the loop task — a raw queue whose producer is the callback (which now does only a memcpy and an index update) and whose consumer is a process_raw_packets function running in the loop task on every tick:

typedef struct { uint8_t bytes[250]; uint16_t len; uint8_t mac[6]; } raw_packet_t;
static raw_packet_t raw_queue[64];   /* depth raised 16 -> 64 in a later revision */
static volatile uint8_t raw_head = 0, raw_tail = 0;
static portMUX_TYPE raw_mux = portMUX_INITIALIZER_UNLOCKED;

void on_data_recv(const uint8_t *mac, const uint8_t *data, int len) {
    /* WiFi-task callback, now minimum-work */
    if (len <= 0 || (size_t)len > sizeof(raw_queue[0].bytes)) return;   /* drop malformed/oversized */
    portENTER_CRITICAL(&raw_mux);
    uint8_t next = (raw_head + 1) % 64;
    if (next != raw_tail) {
        memcpy(raw_queue[raw_head].bytes, data, len);
        raw_queue[raw_head].len = (uint16_t)len;
        memcpy(raw_queue[raw_head].mac, mac, 6);
        raw_head = next;
    }
    portEXIT_CRITICAL(&raw_mux);
}

void process_raw_packets(void) {
    /* called from the main loop on every tick — drains the raw queue,
     * decodes each packet, and writes a reading_t into the original
     * `buffer` shown earlier in this section. */
    ...
}

The original buffer becomes downstream of the new raw queue: process_raw_packets decodes into it and post_readings drains it, both from the loop task. The genuinely cross-task pairing is now only the new one — the raw queue, between the WiFi callback and the loop. buffer's own critical section, load-bearing back when the callback wrote buffer directly, now guards a cross-task write the refactor removed: the same primitive, gone vestigial one revision later.

It is the same lesson, learned a second time. The RX callback runs in the high-priority WiFi task; returning from it hands control back to that task, which resumes draining its own receive queue — and if the callback took more than a few microseconds, the next packet's RX slot can be gone before the radio fills it. Do less per call is the canonical ISR rule, and I had absorbed it for true ISRs but not for high-priority task callbacks — I had filed this one under "task" rather than "ISR-equivalent." Cutting the callback to memcpy-and-return fixed the drops. And portENTER_CRITICAL is still the right call for that new pairing — this time not because an ISR forces it (both ends are tasks) but because the callback's budget is small enough that a mutex's heavier acquire-release could consume a meaningful fraction of it.

Answer two: threading.Thread, and a missing lock that isn't the GIL

This one I built. A CS:APP-style binary bomb phones home to a grading server each time you defuse a phase; to work the lab offline, I wrote an LD_PRELOAD shim that redirects the bomb's gethostbyname to localhost and a small Python server to stand in for the grader. The server is the producer-consumer instance — many connections in, one log out — and it is roughly the following, simplified:

LOGFILE = "submissions.log"

def handle(conn, addr):
    data = b""
    while b"\r\n\r\n" not in data and len(data) < 8192:
        chunk = conn.recv(4096)
        if not chunk:
            break
        data += chunk
    with open(LOGFILE, "ab") as f:        # the only cross-thread shared state
        f.write(b"--- submission ---\n")
        f.write(data)
    conn.sendall(RESPONSE)

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("127.0.0.1", PORT))
server.listen(16)
while True:
    conn, addr = server.accept()
    threading.Thread(target=handle, args=(conn, addr), daemon=True).start()

Every inbound TCP connection produces a Python thread. Each thread reads its request into a local data buffer — not shared — and then appends one record to the log file. The bytes accumulate locally; the only state two threads can touch at the same time is the file on disk. There is no threading.Lock, no queue.Queue, no explicit user-space synchronization primitive in the module — and the obvious suspect for why that is safe, the GIL, is the wrong place to look. Each thread opens its own file object ("ab", which sets O_APPEND on the descriptor), so there is no shared Python object being mutated; the GIL still serializes each thread's bytecode, but that is not what keeps the records apart.

Two cooperating layers keep them apart, and neither is the GIL. For the small requests this server actually handles, CPython's I/O buffering holds both f.write calls and flushes the complete record through one underlying write when the block exits; the kernel's O_APPEND then places that single write at the current end of the file atomically. Neither half suffices alone — O_APPEND makes only an individual write atomic, not an arbitrary sequence of them, so it is the buffering that first coalesces this record into one write. The lock I did not write is split between the interpreter's buffer and the kernel's append semantics: correct without an application-level lock for this CPython build and this workload, and for reasons that would not survive a larger record or a different buffer.

That boundary is exactly the kind of thing that works in testing and fails under load. A submission large enough to flush across several write calls — and the receive loop above does not actually hold one below the buffer size — would split into multiple appends, and then the kernel guarantees only that each write lands at the end, not that the record stays contiguous. At that point the fix is an explicit lock around the whole record, or a single writer draining a queue.Queue. Nothing in the shape of the code tells you where that line is; you have to know what the buffer coalesces and what O_APPEND does and does not promise. An strace shows both sides of it: under four concurrent connections each small record is a single O_APPEND write() and the four land intact and unbroken — out of order, but never spliced together — while a record pushed past the buffer splits into multiple writes, any of which another appender could now interleave.

Had I accumulated into a shared in-memory list instead of a file — connections.append((addr, data)) — the question would move up a layer, into the interpreter, and the answer would change with it. list.append compiles to a single C-level call that runs with the GIL held throughout, so two threads appending to the same list produce both appends, in some order, with no lost update. The atomic unit is the single C-level call the bytecode dispatches into — list_append runs to completion before the interpreter can hand the GIL to another thread — so a single list.append of an already-built value is atomic — as are the other built-in container mutations the CPython FAQ lists, with one caveat: the moment an argument runs its own Python code mid-call (a custom iterable in extend, a __del__ fired by a replaced value), the guarantee is gone. That version would also be correct without an explicit lock, but for a guarantee owned by CPython rather than by the kernel.

And the interpreter's guarantee is narrower than it looks: it does not extend across two such calls. The sequence n = len(shared); shared.append(item) is two C dispatches with interpreter bytecode between them, where the GIL can be handed to another thread; two producers can read the same length and both append, and each one's belief about where its item landed is now wrong. A refactor that splits one atomic call into a read-then-write reintroduces the race with no visible change to the code's structure. The right primitive at that point is threading.Lock around the compound sequence — which is exactly what queue.Queue, the canonical producer-consumer buffer in the standard library, encapsulates so you do not have to.

The whole guarantee belongs to the implementation, not the language — CPython has never promised this atomicity in the spec; it falls out of the GIL. Leave the GIL and it changes character rather than simply vanishing. On a free-threaded build — officially supported in CPython 3.14 under PEP 779, with the GIL genuinely off — list.append is still atomic, but now because the interpreter takes a per-object lock around the call; CPython documents that as current-implementation behavior, not a promise, and tells you to reach for an explicit threading.Lock rather than depend on it. What the per-object lock does not cover is what actually breaks: the len-then-append sequence whose race the GIL only ever made less likely, never impossible; C extensions that assumed one global lock; and mutation through an iterator shared between threads. Jython and IronPython, with no GIL on the JVM and the CLR, draw the line in yet another place. Code that leaned on the GIL for correctness rather than for performance finds out where each runtime draws its line the moment it leaves the one that gave it the guarantee.

Two versions of the same server, then: append to a file, append to a list. They look like the same program, and they are kept correct by different machinery — the file version by CPython's buffer feeding the kernel's O_APPEND, the list version by the interpreter's GIL. The absence of a lock is not the absence of synchronization. It is synchronization someone else wrote, living one or two layers down, and the only way to know which layer owns it is to go and look.

Answer three: composed safety in Go, and the absence of a user-code sync.Mutex

The Go ingest service for the motor-monitoring project is roughly the following, in skeleton:

var (
    db    *pgxpool.Pool
    gauges = map[string]*prometheus.GaugeVec{
        "temp": prometheus.NewGaugeVec(...),
    }
)

func handleReadings(w http.ResponseWriter, r *http.Request) {
    var payload struct{ Readings []reading `json:"readings"` }
    json.NewDecoder(r.Body).Decode(&payload)
    for _, rdg := range payload.Readings {
        _, _ = db.Exec(r.Context(), "INSERT INTO readings ...", rdg.NodeID, rdg.TempC, ...)
        gauges["temp"].WithLabelValues(rdg.NodeID).Set(rdg.TempC)
    }
    w.WriteHeader(http.StatusOK)
}

func main() {
    db, _ = pgxpool.New(ctx, dbURL)
    http.HandleFunc("/api/v1/readings", handleReadings)
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":8080", nil)
}

The ingest receives an HTTP POST every minute from each gateway. Go's net/http server handles each inbound connection in its own service goroutine, and concurrent connections produce concurrent handler invocations — there is no thread pool to share, no explicit dispatch, just goroutines and the runtime's scheduler. The handler reads the request body, iterates over the readings, and for each reading does two things: it inserts a row into Postgres via a pgxpool.Pool, and it calls Set on a prometheus.GaugeVec. Both pgxpool.Pool and prometheus.GaugeVec are package-level shared state, accessed concurrently by every goroutine handling every concurrent request.

There is no sync.Mutex in the user code. There is no sync.RWMutex. There is no chan carrying the readings to a single dispatcher. The handler just calls into the libraries directly, from every goroutine, for every request, with almost no synchronization the user has authored — one small exception, which I will come to.

This is race-safe — concurrency-correct, not fully correct — because the libraries have already done the synchronization on the user's behalf. pgxpool.Pool's Exec method internally acquires a connection from the pool, which involves acquiring a mutex on the pool's internal state, checking out a connection, executing the query, returning the connection. The mutex is in the pgx source code, exactly where it would have to be, and the user never sees it. prometheus.GaugeVec's WithLabelValues method internally acquires a sync.RWMutex on the vector's label-to-gauge map to look up or create the gauge for the given labels, returning a Gauge whose Set method updates a uint64 value field with atomic.StoreUint64 — no second mutex, just a single lock-free atomic store. The two layers (the labels map and the per-gauge value field) are protected by appropriately different primitives in the library's source — a mutex where a map needs one, an atomic where a single word does — and the user never sees either.

The user-code discipline this requires is "never introduce shared state that the libraries already in play do not cover." The handler's local variables — the decoded payload, the loop variables, the JSON decoder, the response writer — are all local to the goroutine and not shared with any other. The shared state is the db pool and the gauges. The pool and each GaugeVec are reached through library APIs that synchronize internally; the outer gauges map is user-owned, but it is built before the server starts and only read afterward, so concurrent lookups are safe by immutability, not by a lock. No user-owned mutable value is accessed concurrently without either synchronization or immutable publication. The concurrent access is inside the libraries, and the libraries synchronize it — a synchronized library has no race to "handle"; it has prevented the one the user code would otherwise have.

There is one exception, and it proves the rule. A later revision added a heartbeat: a gauge reporting how many seconds have passed since the gateway last POSTed, so a dark gateway is visible even when no row is arriving to move any per-reading gauge. That needed exactly one piece of user-authored shared state — a single atomic.Int64, written by the handler once per POST and read by the metrics endpoint on every scrape:

var lastGatewayPostUnix atomic.Int64
// in the handler, once per accepted POST:
lastGatewayPostUnix.Store(time.Now().Unix())
// in a GaugeFunc, on every /metrics scrape:
age := time.Now().Unix() - lastGatewayPostUnix.Load()

Written on each accepted POST, read on every scrape, with no invariant spanning two operations — an atomic load and store is exactly the right primitive (it stays correct even if two POSTs land at once), and a sync.Mutex would be ceremony around a single word. The discipline is not "never author synchronization." It is "author the cheapest primitive the access pattern allows, and delegate everything the libraries already cover."

The discipline is different from the Python in-memory case in one important way. That case relies on the runtime's C-call-level atomicity for user operations; the Go answer relies on the libraries' internal locking for library operations. In Python, you can write code that mutates shared state directly (connections.append(...)) and have it be correct because the language's runtime makes the operation atomic. In Go, you cannot — the language has no GIL and goroutines on a multi-core machine are genuinely concurrent at the hardware level, so a shared-state mutation that CPython's GIL would de-facto serialize is, in Go, a data race you have to synchronize explicitly — one the race detector will flag, but only on a path it actually exercises under -race. The Go discipline is to push the synchronization into libraries that have already paid for it, not to rely on the language's runtime to provide it.

The cost of the Go discipline is that a user who introduces concurrently-accessed mutable shared state without synchronization produces a bug. The race detector finds it only if a test exercises the racing path; production can hide it for a long time. Go's standard library — database/sql, net/http, sync.Map, atomic — and the third-party libraries this handler leans on, the Prometheus client and pgx, are built so that for many common workloads the handler code can avoid authoring its own synchronization, because the libraries it composes already do it. That doesn't make user-code sync.Mutex weird in general; the language's memory model explicitly recommends sync and sync/atomic for serializing shared access, and any program that owns shared mutable state outside a library's coverage will and should use them. The point about this particular handler is narrower: the workload is already covered by libraries, so a user-code mutex would be redundant rather than wrong.

The benefit of the discipline is that the goroutine-per-connection architecture of net/http scales through composition rather than through user-coded synchronization. On the concurrency axis the handler barely has to think at all, because the libraries it calls into handle that — but concurrency-safety is not correctness, and the handler still owns the malformed input, the failed insert, the ordering, and any invariant that spans more than one call. The user-code surface of a Go web service is in some ways closer to the user-code surface of a single-threaded Python program than it is to a user-coded multithreaded C program — because the synchronization is delegated to the libraries, the user code can be written as if it were sequential. The illusion is correct under load only because the libraries are correct under load.

When the consumer fans out

Every answer so far coordinates many concurrent writers at the point where they touch shared state, by excluding competitors or by publishing atomically. The ESP32 excludes competing access to its queue; the Python server leans on one atomic append per record; the Go libraries serialize or atomically update the particular internal state that needs it. None of this makes the whole system single-threaded — the Go service runs its handlers in parallel across pooled connections — but each establishes the small atomicity boundary its shared object requires. That is the right shape when the contention is on the write side. It can become unnecessarily restrictive when the read side fans out.

Two other Go services I work on invert the shape, and the inversion is instructive because they are both Go: the runtime is held fixed, so the only thing moving the primitive is which side of the buffer fans out.

The first is an in-memory aggregator that tails a SQL Server and serves Grafana. A handful of ingester goroutines poll the source tables on a fixed cadence and write pre-aggregated points into typed in-process stores; every Grafana panel request, handled by net/http on its connection's goroutine, reads those same stores. The write side is many-to-one; the read side is one-to-many, and the read side is the axis that scales — fifty operators on shift is fifty concurrent readers of the same store. The primitive is a reader-writer lock:

type Series struct {
    mu      sync.RWMutex
    raw     []Sample
    buckets map[int64]*Welford
}

An RWMutex exists because reads do not need to exclude each other. The ingester takes Lock() for the few microseconds it appends a point; every panel handler takes RLock() and runs concurrently with every other reader, blocking only against the occasional writer. On the ESP32 this primitive would have been pure overhead — there was one reader, the loop, so there was nothing to fan out and an RWMutex's extra bookkeeping would have bought nothing. Same producer-consumer shape, opposite fan direction, and that is what opens the door to a shared read lock instead of an exclusive critical section — the read sections still have to be long enough to be worth the bookkeeping, which the next paragraph gets to. The sleep model did not change; the number of consumers did.

The same service shows the choice is not even binary. A monotonic "newest timestamp seen" cache, touched on every write, uses a plain sync.Mutex, not an RWMutex — its critical section is a single comparison, so there is nothing for a read lock to amortize and the reader-writer bookkeeping would cost more than it saves. An ingester's high-water-mark cursor, one int64 read by an unrelated goroutine, uses atomic.Int64 — one word, no compound invariant. The primitives form a ladder, and the rung is chosen by how much work happens under the lock and how the readers fan out, not by habit.

The same service answers a second one-to-many problem, in a different subsystem, and the answer is the inverse of the read lock. Some routes are too expensive to pre-aggregate in memory, so they are served straight from SQL behind a TTL cache — and a cold miss on a popular panel can be requested by a dozen Grafana goroutines in the same instant. Letting all twelve run the five-second scan concurrently is precisely what the RWMutex would permit and precisely what you do not want here. So the cache collapses them: the first goroutine to miss registers a channel and becomes the sole fetcher; the rest find that channel and park on it.

// the goroutine that wins the race fetches once, then broadcasts:
ch := make(chan struct{})
c.inflight[key] = ch
c.mu.Unlock()
rows, err := fetch()                 // one SQL scan, shared by every waiter
c.mu.Lock()
c.entries[key] = &entry{rows: rows, err: err, fetchedAt: time.Now(), ttl: ttl}
delete(c.inflight, key)
close(ch)                            // wakes every goroutine blocked on <-ch
c.mu.Unlock()

The Mutex here guards only the two maps; the channel does the real work, and it does it as a broadcast. close(ch) is the Go idiom for releasing an unbounded set of waiters at once — every goroutine blocked on <-ch unblocks the instant the channel closes, then re-reads the entry the leader just published. A dozen concurrent cold-miss requests become one SQL scan and eleven memory reads. The RWMutex lets readers run in parallel; the single-flight makes them converge on a single result and then fan back out through the close. Same one-to-many problem, opposite tactic, a different primitive again — a channel, not a lock.

The collapse holds on failure, too — which is the part you only learn by reading the function to its end. The leader publishes whatever fetch returned, rows or error, before it closes the channel, so every waiter reads that one result with the error included; and because the entry is fresh for its TTL, a failed fetch is negative-cached for the whole window, the next caller getting the stored error rather than restampeding the database. The genuine hazard is narrower and nastier: if fetch panics, the leader never reaches the publish-and-close, the channel stays open, and every waiter blocks on it forever with the key wedged in the inflight map. The fix is a defer that closes the channel and clears the key — it would run as the panic unwinds, no recover required; the version above just doesn't have one. The happy path tells you none of this — the control flow does.

The second service goes further and removes the primitive entirely. My blog loads every post and template off disk once, in the server constructor, before it accepts a single connection — then serves them to an unbounded number of request goroutines with no lock of any kind:

s.posts, err = generator.LoadAllPosts("content/posts")
// New() returns here; only afterward does the server start accepting connections.

The safety is structural, not synchronized. Every write happens-before any reader exists, so no reader can observe a torn or half-built value; the slices, and the structs they point at, are never mutated again. When shared state is written once before any reader exists and never after, the correct concurrency primitive is the absence of one. The same file holds exactly one real lock — a plain Mutex around a rate-limiter map — and it is a plain Mutex, not an RWMutex, because there every access is a write. Read-mostly usually wants RWMutex; write-mostly wants Mutex; write-once wants nothing. One file, one runtime, three answers, selected by fan direction, mutation rate, and how much work each access does under the lock.

That last claim has a boundary, and naming it is the same discipline as everything else here: the immutability is absolute only because there is no hot-reload. Add a SIGHUP handler or a filesystem watcher that swaps the posts under live readers and the guarantee evaporates — you would need to publish the new set through an atomic.Value or behind an RWMutex. "Written once" is a property of the lifecycle, not of the type, and the lifecycle is the thing you have to keep true.

What the comparison surfaces

Three runtimes, three answers, no two of them interchangeable — and then a fourth observation that cuts across all three: flip the consumer from one to many, and the answer changes again without the runtime changing at all.

The ESP32 firmware uses a critical section because the producer (the ESP-NOW receive callback in the WiFi task) and the consumer (the main loop in the app task) live in different FreeRTOS tasks that can be scheduled in either order, with the WiFi task documented as a context whose callbacks should do minimal work. The inbound handoff between them needs mutual exclusion. A FreeRTOS mutex would satisfy the contract — both contexts are tasks that can sleep — but the critical section is short, and a mutex can block and carries heavier RTOS bookkeeping where this does not. portENTER_CRITICAL is the cheapest mechanism: it raises the interrupt level on the running core — suppressing ordinary ISRs and task preemption, since the scheduler runs from a timer interrupt — and, on the dual-core part, acquires a spinlock against the other core, all without invoking the scheduler. The runtime's sleep model — "tasks can sleep, so a mutex can block and rides the scheduler's wait machinery, while raising the interrupt level suppresses preemption for the price of a few instructions" — dictates the primitive.

The Python server uses no explicit lock because CPython's buffering coalesces each record into one write and the kernel's O_APPEND lands it atomically — the guarantee is split between the interpreter's buffer and the kernel, not the GIL you would check first. (Had the shared state been an in-memory list, it would have been correct without a lock for a different reason: list.append is a single GIL-held C call. Different machinery, near-identical code.) The sleep model's lesson here is to find which layer owns the guarantee before trusting it.

The Go service uses no user-code lock because its shared state is mediated by libraries that synchronize internally. The runtime's sleep model — "goroutines may run concurrently on different cores, the language has no implicit serialization, synchronization is the libraries' responsibility unless the user accepts the burden of sync.Mutex themselves" — dictates the discipline of "compose synchronization through libraries."

In each case, the right primitive is not "the one I would have picked from a survey of concurrency primitives." It is the one that satisfies the runtime's constraints for the producer-consumer shape. Substituting one runtime's primitive for another's would produce code that is wrong in subtle ways. A portENTER_CRITICAL macro inside a Python program would do nothing useful, because Python does not have ISRs and does not support disabling interrupts. A blocking FreeRTOS mutex called from inside an ISR would be a documented mistake, because ISRs cannot sleep on a wait — FreeRTOS provides separate ISR-safe APIs for exactly that reason. A sync.Mutex inside this specific goroutine handler would be redundant for a workload that already goes through pgxpool and prometheus.GaugeVec, and would slow down the handler with no correctness benefit — though sync.Mutex is a perfectly normal primitive in Go user code anywhere the program owns shared mutable state that no library is already covering.

The right question is not "which primitive do I use." It is two questions: "what is my runtime's sleep model, and what does it make cheap or free?" and "which side of the buffer fans out?" Once those are clear, the primitive largely falls out of them — an exclusive barrier when the consumer is alone, a shared read lock when the consumer fans out, a broadcast channel when a stampede of readers should collapse onto one fetch, an atomic when the shared state is a single word, nothing at all when the writer finishes before any reader exists. The primitive's name is a portability detail; the primitive's mechanism is what matters at the chip-and-instruction level. The contract that the primitive's name describes is the portable thing.

What this teaches about reading other people's concurrent code

There is a habit I would like to recommend, on the strength of having had to develop it, which is to never read a primitive's name without also reading its mechanism. The names "spinlock," "mutex," "semaphore," "atomic," and "barrier" are conventional labels for contracts that have specific meanings in specific runtimes. The same name in two different runtimes can mean two completely different things. "Spinlock" in the Linux kernel, on a typical non-RT SMP build, disables preemption and uses architecture-specific atomic operations and memory barriers — and under PREEMPT_RT the ordinary spinlock_t becomes a sleeping RT-mutex while raw_spinlock_t keeps spinning. "Spinlock" in FreeRTOS on a dual-core chip is a similar thing with a different ABI. "Spinlock" in FreeRTOS on a single-core chip compiles to no spin at all — the lock degrades to a no-op and the mutual exclusion comes from raising the core's interrupt level. The three are not the same mechanism. They satisfy related but not identical contracts. Reading them as if they were the same is the kind of mistake that produces bugs that look like "well, the literature says spinlocks are fast, so why is my code slow."

The discipline I have come to is to read the source of the primitive when I encounter it for the first time in any new runtime. The source is the only thing that tells me what the primitive actually does. The documentation tells me what the primitive is supposed to do. The two are usually the same; they are not always; the difference is the kind of thing that is worth a few minutes of source-reading to notice.

For this post, I read the ESP-IDF's FreeRTOS ports to confirm that on the single-core part portENTER_CRITICAL raises the interrupt level with the spinlock compiled out, and on the dual-core part it adds a real cross-core spinlock. I read the POSIX specification of O_APPEND to confirm that the atomicity the bomb-lab server relies on is the kernel's and not the GIL's. I read CPython's list_append to confirm it holds the GIL across the whole C call. I read pgx's pool source to confirm that Exec acquires the pool's mutex. More than one of these readings corrected something I would otherwise have stated wrong, and the act of reading was the part that calibrated my confidence. Without the readings, I would have been working from documentation, which is mostly accurate, and from blog posts, which are sometimes accurate, and from my own intuitions, which are calibrated against runtimes I do not currently work in. The source tells you what this version actually does; the documentation tells you what you are allowed to depend on. You want both, and you want to know which is which.

The closing observation

If I had to compress this post to one sentence — which I will not, because the comparison is the post — the sentence would be: concurrency primitives are not a menu you pick from; they are primarily a function of two things — the runtime's sleep model and which side of the buffer fans out — and the right primitive largely falls out of those rather than from the primitive's name.

The three answers are correct in their respective runtimes. None of them transfers to another unchanged.

This is the kind of comparison that becomes available only when you have worked across multiple runtimes solving the same shape of problem. In my case, two of the three were the producer and consumer of one project's data pipeline — the same shape of problem encountered twice within one project, on opposite sides of an HTTPS boundary, requiring two completely different answers. The third was from an unrelated CTF codebase that happened to have the same shape too. The comparison was not available to me before I had reason to look at the same problem in three places, and it is, in retrospect, the kind of comparison I wish the surveys foregrounded — because the surveys tend to enumerate primitives without grounding them in the runtime models that determine which primitive is the right one. The right primitive is not portable. The contract is. Read the documentation for the contract — the part you can carry between runtimes — and read the source to see how this one implements it, and whether the two still agree.

The last thing I want to note, which is the kind of thing the rest of the post would be padded with bullets if I wanted to trail off, is that I used to find the proliferation of concurrency primitives across runtimes confusing. I no longer do, because I have stopped trying to map them onto each other. Each runtime has its own primitives, named for legacy reasons that may or may not be illuminating, mechanized for the runtime's specific sleep model, and correct within that model. There is no universal concurrency primitive. There is no one-size-fits-all answer. There is, instead, a small set of locally-correct answers, each derived from the runtime's sleep model, and the engineer's job is to know the sleep model well enough to pick the locally-correct answer. The question — what is this runtime's sleep model? — is the portable thing; the answer never is.

If you read one paragraph of this post and forget the rest, let it be that one.