Search for a command to run...
SuperSearch started failing under load with an error that looked like a capacity problem. A request would get rejected with a tokens-per-minute limit hit, and the obvious reading was that we were asking the model to do too much — too much context, too many tools, too much thinking. We went looking for what to cut.
The actual cause was the opposite. We were not using too much. We were reserving too much, and the reservation itself was what got us rejected. Here is the number that untangled it.
The failure looked like this:
TPM limit: 200000
Already used: 165003
Requested: 47793
A tokens-per-minute admission failure. The organization had a rolling limit of two hundred thousand tokens per minute, a good chunk was already in use, and this one request was asking for nearly forty-eight thousand more. Add them up and you cross the line, so the request was turned away.
The question was: forty-eight thousand tokens for what? A single agent step does not generate forty-eight thousand tokens of output. An investigation reflection is a few hundred words. Even a full synthesis answer is a couple of thousand. Where was the rest coming from?
The model we use, gpt-4.1-mini, allows up to 32,768 tokens of output on a single request. That is the ceiling — the most the model is permitted to generate in response. And here is the part I had not internalized: the rate limiter does not wait to see how much you actually generate. It estimates your usage up front, from your input size plus your configured maximum output size, and reserves that estimate against your limit before the model writes a single token.
We had never set a maximum output size. Which meant every request was implicitly telling the system: I might generate up to 32,768 tokens. Hold that much capacity for me. Just in case.
Now the arithmetic snaps into place. The request had about 15,025 tokens of input — the prompt, the tools, the conversation so far. Add the 32,768 output tokens we were reserving by default:
15,025 input + 32,768 reserved output = 47,793
That is exactly the requested figure. The overwhelming majority of what we were asking for was output capacity we had no intention of using. An investigation reflection needs maybe two thousand tokens. We were reserving sixteen times that, on every request, on every step, for an answer that was never going to come.
The fix was a few lines. Set explicit output caps, per phase.
An investigation step — a plan or a reflection plus maybe a tool call — gets two thousand tokens. That is generous for what it actually writes. The final synthesis answer gets four thousand, enough for a real research response with room to breathe. An empty fallback gets two hundred. These caps are shared by the live route and the evaluation harness, so a fix in one cannot drift from the other.
Now the reservation for that same request:
Before: 15,025 input + 32,768 reserved = 47,793
After: 15,025 input + 2,000 reserved = 17,025
A 64 percent reduction in the per-step capacity we were claiming, without changing a single thing about what the model actually reads or writes. We were not throttling the agent. We were stopping pretending it might write a novella.
The deeper lesson is about defaults.
When you do not specify a limit, the system does not pick nothing. It picks the maximum, because the maximum is the safe choice from the system's perspective — it guarantees the model will never be cut off. But "safe for the model" and "responsible for your rate limit" are different goals, and the default optimizes for the first at the expense of the second. Every unset knob is not a zero. It is a silent vote for the most expensive option.
This is everywhere once you look for it. Unset cache sizes default to unbounded. Unset timeouts default to forever. Unset output limits default to the ceiling. In each case the system is protecting you from one failure mode (running out of space, cutting off early) by quietly exposing you to another (using everything, hanging forever, reserving the whole budget). The protection is real. The cost is invisible until you trip over it.
We found this one because a request got rejected and the error had a number in it. I wonder how many others are still running, silently reserving their own thirty-two thousand tokens, and nobody has noticed because the limit has not been hit yet.
Set your limits. The defaults are not free.