The model swap that would have failed without a sound

Stolperfalle is the knowledge base my coding agents write into and read from. When a session runs into a trap, reflect turns the session summary into candidate knowledge units (KUs), and the agent proposes the good ones. Hooks then query the store whenever a prompt or a failed shell command carries a real error signal (a traceback, an exit code, an HTTP status), and put the best match straight into the agent’s context. So a KU doesn’t sit in a database somewhere; it lands in a live session.

Until this week reflect ran on gpt-4o-mini. OpenAI shuts that model down on 14 January, so on Friday 23 October I sat down to pick its replacement.

The swap that would have failed without a sound

The obvious move was one env var: point it at OpenAI’s named successor, gpt-6-luna. I tried that first, with the exact request we send in production. Every call came back HTTP 400, because luna is a reasoning model and doesn’t accept temperature.

That alone would’ve been fine. What hurt was what reflect did with the error: log a warning and fall back to a keyword heuristic. Its response even said method: heuristic, but nothing watched for it. So the tool would’ve kept answering, just a lot worse, until someone noticed the knowledge units getting vague.

That part is fixed now. A 400, 401, 403 or 404 from the provider means the config is wrong, so it raises an error instead of falling back. Timeouts and 5xx still fall back, but they log as errors, say why in the response, and show up as a count in status.

Twelve cases, one blind grader

So I built a small eval instead of guessing. Twelve session summaries, made from fifteen real KUs in the store: nine with one or two real learnings buried in project noise (“renamed an env var”, “picked port 3001”) and three with nothing worth keeping. One is in German, and two use the terse format our Stop hook sends. All models got the same prompt text; the Claude models answered through a JSON schema, the OpenAI ones through the old free-form JSON. A separate Claude agent graded all 72 outputs blind, with the labels shuffled per case.

gpt-4o-mini came last. Its candidates were short and generic, and they dropped the product names and versions that make a KU findable later. Every Claude model found all 13 expected learnings. Sonnet 5.5 at low effort had the fewest bad candidates (one) and the best overall score, 4.67 out of 5. Haiku was close on everything except that overall score, and the eval doesn’t separate the two cleanly. I went with Sonnet because the difference is about a cent a call. Opus wasn’t better, at twice Sonnet’s price. The whole comparison cost about 36 cents.

A cent per reflect call is roughly 47 times what gpt-4o-mini cost. Even at a hundred calls a day that stays under 30 euros a month.

Two problems showed up in every model, and those were prompt fixes rather than model fixes. They all happily extracted a bug in the project’s own code as a “general learning”, and some answered in German when the summary was German. The shipped prompt handles both. I re-ran all twelve cases on the exact code that went to production, and checked a live call after the deploy.

The plan had a second grader from outside Anthropic (Codex), but its login had expired, so a Claude agent graded Claude output, and it may well prefer it. I’m treating the gaps between rows as a hint, not a measurement. The gap to gpt-4o-mini is big enough that I acted on it, but it isn’t controlled for that bias.

While I was in there

The swap made me curious about what was already stored. Four reviewer agents split 599 of the roughly 640 live units between them, each with an index of the whole store to catch duplicates across their slices. Claude agents did the reviewing, the fact-checks and the rewrites; I decided what got applied.

Most of it holds up, as far as a model can judge: 527 (88%) stayed as they were. 32 duplicates got merged. 26 KUs needed a rewrite (client names, internal hostnames, several traps bundled under one summary, leftover tool-call markup), and 22 of those went through. Ten looked factually wrong and went to an agent that checked them against docs, specs and source. Nine were wrong in a way that mattered, like a Docker KU claiming ${VAR} interpolation keeps secrets out of docker inspect (it doesn’t) or a zsh one saying unquoted $(cmd) isn’t word-split (in zsh 5.9 it is). One was right and the reviewer was wrong; it reproduced on macOS’s bash 3.2. Three trivial ones were marked stale. The six left in place are the four rewrites that bounced, one pair of near-duplicates I kept on purpose, and the one that was right.

The patterns are more useful than the fixes:

• 94% of the store sits at the default confidence with zero confirmations. Agents query and use KUs all day and almost never call confirm, so confidence doesn’t tell you much yet. It probably has to come from what agents do (a KU was injected, then the command worked) rather than from asking them to vote.
• Twelve separate KUs say roughly the same thing about zsh versus bash. Agents keep relearning it, and the duplicate check at propose time misses rephrasings.
• There’s no edit. Fixing a KU means proposing a new one and superseding the old, and four of my 26 rewrites bounced because the fix was too close to the original. The check that lets rephrasings through also blocks corrections. Tsja.
• Client names and hostnames got in because nothing redacts at propose time. The store runs on my own server for my own agents, so it stayed in the house, but before another team writes into one of these, redaction has to happen on the way in. And reflect now sends session summaries to Anthropic, which is one more processor to name in a contract.

Next up is the boring part: an edit path, redaction on the way in, and a confirm signal that comes from use.

The code is public in mcp-stolperfalle, and the comparison is written up in docs/reflect-model-eval.md. If you’re deciding whether to run it for a team, start with the one-page overview: what data leaves the server, isolation, cost, the gaps above, and how you’d leave again.