What broke

The README said agents had budgets. They did, in the sense that spend was recorded. When I wrote a test that pushed an agent over its daily limit and counted the calls reaching the model, the call went straight through: 200 where I expected 429.

The policy function returned None every time, and nothing on the request path called it anyway. Its unit tests passed because they only tested the function by itself.

The fix reserves an estimated cost before forwarding, against the agent, run and global limits at once, then corrects it to the real token count afterwards. If the budget store is down, the call is refused. These days my tests count what actually reaches the upstream.

I wanted to know whether 'every request is audited' survives a crash. The harness sends traffic, kills the proxy with kill -9 mid-load, restarts it, and checks every request the client got a 200 for against the database.

At 600 req/s, 3 of 30 runs lost one event each. The response event was written from a background task, so if the kill landed in the millisecond after the client got its answer, the event was gone. The same runs turned up a second bug: 11 of them failed hash-chain verification, because two requests starting at once on a new session each picked a different first link.

Now the event goes into a write-ahead journal before the client is answered, and the chain's first link comes from the session id. Before this, a full queue just dropped events and logged a warning. I had been calling that 'fail open'.

Before Govrix pulls documents into a prompt, it checks that the agent owns the bucket. Reading that code during the audit, I found that a timeout or a 500 from the ownership lookup counted as yes.

I'd made 'fail open' the default for the audit path, where it's the right call, and it had crept into a permission check, where it means one agent can read another agent's documents during an outage.

Now an unanswered check is a no. The one exception is a 404 just after a bucket is created, because of a create-then-ingest race, and that exception is written down next to the code.

With memory injection on, vLLM's prefix-cache hit rate over a 20-turn chat fell to 9%. Going straight to vLLM without the proxy, it was 95%.

I was adding retrieved context as a new system message at the very top of the prompt. That changed the first bytes on every turn, so nothing after them could be reused. Re-serialising the JSON body also reordered its keys: the same request, different bytes.

Now the context goes at the start of the last user message, spliced in as raw bytes, and the rest of the request passes through untouched.

One day Crucible couldn't create a sandbox. Docker said 'all predefined address pools have been fully subnetted'. Every sandbox gets its own container and network, nothing ever deleted them, and 29 had piled up.

Pausing a sandbox also reset its idle timer, so paused ones would never have expired even with a cleanup job.

I added a reaper. It clears orphans when the service starts, expires idle sandboxes every 30 seconds and caps how many can exist. Review caught a thread race in my first version.

The first load test crashed the proxy at about 1,300 users with 'Too many open files'. That one was easy: the container's file limit was 2,048.

After raising it, errors started near 2,500 users, and I wrote that down as the proxy's limit. It wasn't. A single Locust process was saturating, and a bug in my failure handler had hidden 55,185 exceptions.

With 4 Locust workers the errors went away. I retracted the 2,500 number in its own commit.

Four vLLM replicas at a quarter of the GPU each should fit on a 32 GB card. Three started. The fourth ran out of memory at 31,955 of 32,607 MiB.

Every replica pays a fixed CUDA overhead that the fraction ignores, and the desktop itself was holding 2.3 to 2.75 GiB. While digging, I also found a KV-cache hook that reported success but that vLLM never called, and an offload function that quietly did nothing.

I run two replicas at 0.4 now. The fake hook and the silent offload now fail loudly instead.