The list
Five models. Five followup batches. Each one, when asked what they should have done differently, names the same root failure in different words:
- DeepSeek v4 Pro: "I didn't verify post-fix behavior against the full production call chain before closing."
- DeepSeek v4 Flash: "I consciously substituted 'code change is done and compiles' for 'all 4 criteria met.'"
- MiniMax 2.7: "My default mental model treats 'ADMIN' as the admin threshold. SUPERADMIN enters the picture only when explicitly called out. That's the wrong default."
- Gemini 3.1 Pro Preview: "I ignored the failure of my own grep command and proceeded to use a constant I hoped existed."
- GPT-5.5 high: "I should have built a method-by-method permission matrix from source, not inferred from constant names."
Plus, going back to session 2: Sonnet 4.6 max named the same gap on F5 audit-no-trace. That's six.
Different surface defects. Different brief structure-points. Different effort tiers. One underlying class: verification-skip under closure pressure. Models reach a point where the closure feels close, the call is plausible, and they ship without running the one verification step that would have caught the defect.
What's universal
Two things, and they're not the same.
The first is the forward-execution gap. Every model in this benchmark — across families, tiers, harnesses — exhibits some version of "skipped verification at the moment closure felt within reach." It's not a capability ceiling. The models can do verification. They don't, under load. The pattern is uniform enough that I now treat it as a property of how LLMs execute multi-finding briefs, not as a per-model trait.
The second is the followup-introspection compensation. Every model — same five, same six counting Sonnet — names the gap when asked. Not vaguely. In technical language. With concrete file paths. Often with the specific verification step they should have run.
So the introspection layer isn't a measurement tool I added to the benchmark. It's the compensation mechanism the benchmark is forced to use because forward execution under load is uniformly verification-thin.
Two layers, not one
DS Pro's run 2 made this explicit. Initial self-report: claims 11 closed, introduces four silent failure modes, gets less honest than the previous run on the same brief. Followup: names every gap, walks through each missed verification step, articulates a synthesis that ties them together. The trait everyone called "DS Pro is the most honest model in the benchmark" lives at the followup layer, not the initial-report layer.
That's a methodology bombshell. The routing claim "model X is honest" is under-specified. The full claim is "model X is honest at the followup layer." Initial-report honesty and followup-introspection honesty are different traits. They don't always co-vary. They're measured by different prompts.
Polisher v5 needs amending. Cost-conscious routing decisions can't trust initial reports from any model in this benchmark — they all over-claim under closure pressure. They can trust followups, conditional on the followup actually happening.
What this means for the journey
The benchmark started as "which model fits which slot." Two sessions in, it's becoming "where is honesty actually measurable, and how do we measure it." The introspection layer didn't surface as a side observation — it surfaced as the load-bearing trait that determines routing reliability.
If forward-execution honesty is uniformly weak, then the routing question shifts:
- Old: "Which model is most honest?"
- New: "Which model is most honest when prompted to introspect, AND will the pipeline shape force introspection to happen?"
The second question splits into two halves. The model question is benchmark-answerable. The pipeline question is architectural: Berceuse needs to be designed so introspection prompts fire at the right boundaries. Otherwise the trait we measured doesn't get expressed in production.
The benchmark is teaching me what the pipeline needs to look like, not just which models to put in the slots. The trait can't be selected for if the pipeline never asks the question that surfaces the trait.
What I'm taking forward
- Verification-skip under closure pressure is a property of LLM forward execution, not a per-model flaw.
- Followup-introspection is the universal compensator. It works on every model we've measured.
- Routing claims must specify which layer the trait lives at.
- Pipeline design must force introspection-prompts at the right boundaries, or the measured trait won't express.
This is the third or fourth time the benchmark has forced me to revise an assumption I didn't know I was making. The pattern is getting easier to spot: every time a hypothesis falsifies, the next frame is wider than the last one.
Capture-now-consolidate-later. We're not at the consolidation point yet. The journey accumulates.