A limit measured against a different model

Today the DM moved to a different model.

We shipped it. Turns were fine. Starting a new game became impossible.

What actually broke

From the backend log:

app.dm.OutputTruncated: the DM could not produce a valid narrate_turn call in 3 attempts:
  attempt 1: the narrate_turn call was cut off at the 3000-token cap
  attempt 2: the narrate_turn call was cut off at the 3000-token cap
  attempt 3: the narrate_turn call was cut off at the 3000-token cap

The model was running into the output ceiling. Three times over, nine thousand tokens, 331 seconds — and not one line of it reached the player. The proxy gave up first, at the two-minute mark, so what a person saw was “The server didn’t answer in time” and nothing else. The real reason reached neither them nor us.

Why the opening specifically

Opening a session is the largest single answer the model writes in the whole game.

On an ordinary turn it returns prose, state changes and the choices on offer. An opening adds a scene to that: the opening=True flag moves the scene field from optional to required (contract.py:2643). That is deliberate — it is the one turn with no previous picture to fall back on.

And the output limit on that call was the shared one. The same MAX_TOKENS sized for an ordinary turn (dm.py:828). Three thousand tokens.

It worked, too, for a long time. By our own records the longest opening across 58 sessions on one model was 878 tokens, and across 73 sessions on another it was 1,253. Three thousand covered all of them three times over.

The constant was not a mistake. It was correct for exactly the models it had once been measured against.

What we did about it

The opening got a limit of its own — twice an ordinary turn, and derived from it rather than written down as a separate number:

OPENING_MAX_TOKENS = MAX_TOKENS * 2

A separate constant would have drifted from the one it is supposed to depend on sooner or later. This way a single switch moves both at once, which matters when you raise the limit temporarily to measure a candidate model fairly.

The second change was to the retries. When an answer is cut off by the limit there is nothing to correct: the model did not get it wrong, it ran out of room. The old code asked the identical question at the identical limit three times in a row. If the overrun is a matter of luck that is a cheap bet and it usually wins. If it is systematic, it is three identical failures at the price of three calls and three times the wait. On this call there is now one attempt.

And that was not the whole of it

While we were fixing the limit, a second fault surfaced that the first had been standing in front of the whole time: the provider was returning 429. We were sitting in a shared quota, split between everyone using the router, and at peak hours there was simply not enough of it.

Two separate failures, and for a while each looked like the other. What helped was a detail that is easy to miss: a refusal on quota is not billed, so it writes no row in the usage ledger. In the log an outage of that kind looks like silence rather than a spike. An empty ledger does not mean nobody played — it can mean nobody got an answer.

Code does not fix that. Your own provider key does: the shared queue stops being your queue.

The part worth keeping

A number in the code is not correct on its own. It is correct relative to whatever it was measured against. For as long as the model never changed there was no difference between “correct” and “correct for the model we were on”. The difference appeared on the exact day the model did change — and it appeared not as a warning, but as a game that would not start.