"ok" was a lie

The bug that started it

An orchestrator sends a worker its next assignment. The API answers { ok: true, accepted: true }. The worker never does the work. Nobody errors, nobody retries, nothing is logged as wrong. The orchestrator waits for a result that is never coming, and the whole run quietly stalls.

The message was in the inbox. It just wasn't going anywhere, because the recipient had no terminal behind it, no active wait, and no channel — nothing that would ever cause it to look. accepted was true in the only sense we had implemented: we had accepted it. The sender read it as "delivered", which is the only sense they cared about.

That's the whole post, really. Everything below is us learning it properly.

Storage acknowledgement is not delivery acknowledgement

Our inbox is a durable FIFO queue, and writing to it always succeeds. So ok was structurally incapable of being false, which makes it structurally incapable of carrying information. It was a receipt for the wrong event.

The sender has exactly one question: will anything cause this to be seen? That's now a field of its own. delivered is true when a waiter consumed the message, the channel pushed it, or the terminal typed it — three different routes that all end with the recipient encountering the text. It's false in exactly one case, inbox_only: no waiter, no channel, no live terminal. Then it sits unread until the recipient thinks to poll.

And when it's false, the response carries a warning in plain language — that the recipient has no terminal and nothing will wake it, that the message stays put until it calls inbox or wait, and that you should not block on an answer. The reader of that field is a language model. Writing "inbox_only: true" and expecting the right inference is optimistic; writing the consequence and the way out is not.

The field we deleted twice

The original response also had delivered_via_channel, a boolean reporting whether the SSE push had fired. It is, narrowly, a true fact. It was also the single most misread thing we have ever shipped.

Look at when it's false. A waiter grabbed the message the instant it arrived: false. The terminal typed it straight into the agent: false. Those are the best possible outcomes. The flag reports one sub-route, but its name reads like a verdict, so false next to a perfectly successful send was pure ambiguity — and readers acted on it. We have an internal recovery note that had to be written down after an incident, and one of its rules is literally never infer non-start from delivered_via_channel:false. When you need a written rule telling people how not to be misled by a field, the field is the bug.

So we removed it from the send response and replaced it with delivery_path, which names what actually happened — waiter_and_inbox, sse_channel_and_inbox, terminal_or_queued_and_inbox, inbox_only — and subsumes the SSE case rather than singling it out.

Then we found it was still there. It lived on the stored message too, so it kept surfacing on inbox and wait — the same trap, now aimed at the recipient, who is by definition already holding the message the flag describes. It's now #[serde(skip)]: still recorded server-side for forensics, present in the log line an operator reads, and gone from every response. Removing a field twice from two different responses is a good sign the original concept was wrong, not just its placement.

"The session exists" is not a receipt either

The same mistake was hiding one layer down. When we type a message into a recipient's terminal, that call used to report whether the session existed. Existence is not delivery: a busy agent's composer merely parks the text until the current turn ends.

The consequences composed badly. Every call site marked the message as terminal-dispatched, and messages owned by the terminal are hidden from waiters — so a message that had never actually been typed was simultaneously not the waiter's problem and not sent. It fell between two layers, both of which believed the other had it.

PTY delivery now reports what it did: typed, queued, or unavailable. A queued message stays deliberately pending, visible to a waiter, until it really goes in.

Who is "the recipient", exactly?

A related failure came from one identifier doing two jobs. A peer's identity was both its mailbox address and the handle for its terminal, on the assumption that the two are equal. They never were: PTYs are keyed by a freshly minted UUID while the caller-supplied session id is exported separately.

The visible symptom was absurd — an orchestrator running inside a TUICommander tab could not be woken by its own subagents. Every send came back inbox-only while still answering ok, and mail piled up unread a few centimetres from someone watching the screen.

Delivery now resolves an announced identity to whatever PTY currently backs it, per call. Respawn a terminal and messages follow it without re-registration. Four of the five session-creating paths were injecting no identity at all; all five bind one now. And teardown retires the registration, closing a leak where a self-registered peer outlived its terminal for the rest of the process lifetime.

Don't interrupt the orchestrator

The last category isn't about lost messages — it's about messages that arrive too well.

Typing into an agent's terminal is a genuinely great delivery mechanism: the agent picks the work up on its next turn without polling. It's also an interruption. Do it to a parent that is mid-thought and you've injected a child's result into the middle of its reasoning, steering it away from what it was doing.

So the rules for an orchestrator are deliberately different. An active wait owns delivery when there is one. Otherwise a parent that is authoritatively idle or completed gets a single coalesced, payload-free notice: you have mail, call inbox. Working, awaiting, starting, missing and unknown all fail closed to inbox-only. Failing closed matters — "unknown state" resolving to "go ahead and type into it" is how you corrupt a run you can't reproduce.

Even the retries are bounded. A payload-free wake whose terminal write was uncertain used to expire every five seconds and retry forever. Each unread group now gets its initial wake plus at most one retry; after that the mail stays in the inbox until a real read clears it. An infinite retry loop against an uncertain signal is just a slower way of interrupting someone constantly.

What we'd do differently

Every one of these bugs has the same shape: a field that describes what our code did, presented where the reader wanted to know what happened to the message. ok described a queue write. delivered_via_channel described a branch we took. "Session exists" described a lookup. All three were true statements, and all three were read as delivery receipts, because that's the question the caller had.

Two things we'd carry into any system like this:

Name fields after the reader's question, not the implementation's answer. If the caller wants to know "will this be seen", ship delivered. A field whose correct interpretation needs a paragraph of documentation will be misinterpreted, and no amount of docs fixes it.

Remember the client is a language model. It won't cross-reference four fields to reconstruct your intent — it reads the one whose name matches its question and acts. That makes ambiguity in an agent-facing API more dangerous than in a human-facing one: a human gets confused and asks; a model gets confident and proceeds.