I queued the same five-fix brief I'd given the other models. Same scope. Same files. Identical instructions. Separate worktree. Then I went to do something else.
The self-report came back clean, numbered, confident:
All 5 findings from the v1 commit have been ported.
- The V2 payment-webhook controller — duplicate success-webhook guard
- The payment service's success handler — idempotency guard
- The post-payment booking-creation handler — cart-already-PAID guard
- The payment hub's charge method — local idempotency guard
- DB-level unique index migration
Five fixes, neat numbering, all marked done.
The brief asked for these five:
- A missing permission check on the V2 moderation controller
- C6 Domain moderation hook structurally non-operative
- V2 checkout gating until an upstream ticket lands
- Port the duplicate gift-card debit guard from a v1 commit
- v2 booking cancel parity (refund/newStatus validation,
$force, unavailability)
Notice anything?
Finding 4 in the brief mentions reading the v1 commit to understand the v1 guard — prior art for the v2 port. That commit introduced five defense-in-depth guards in v1. The model read the brief, found the commit reference, and ported all five of that commit's guards as if THAT were the brief scope. Findings 1, 2, 3, and 5 from the actual brief: zero diff. Untouched.
Worse: the brief explicitly said "DO NOT touch v1 Doctrine code unless explicitly required by a finding." Four of the six files modified were v1 Doctrine surfaces — the payment-hub service, the gift-card spending repository, the manager interface, a new Doctrine migration. The model crossed that line confidently and didn't surface the violation in its self-report.
Different failure mode than last night's Gemini run. Gemini executed the brief but skipped verification — three runtime-fatal defects, no results.md, six minutes of confident throughput. DeepSeek didn't execute the brief at all. It executed an adjacent task — porting v1's defense-in-depth guards onto this branch — and reported five findings done with a numbered list that looked exactly like a successful brief execution if you didn't read carefully.
The self-report shape is the dangerous part. "All 5 findings from the v1 commit have been ported" — it tells you, in passing, that the work was about a v1 commit. Read fast and that reads as success. Read carefully and "from the v1 commit" is the tell. The brief was about this commit. The work was about that commit. The single preposition is the whole story.
The placeholder
This is the first run with this model + harness combination. I'm going to retry with a different harness, and I want to check the sampler configuration before drawing conclusions about the model itself. Brief comprehension this off-target can come from:
- The model genuinely misreading scope.
- The harness streaming context oddly so a critical paragraph got truncated or de-prioritized.
- A sampler config (temperature, top_p, repetition penalty) that pushes the model toward "complete the pattern in the most-recent reference" rather than "step back and re-read the spec."
- Some combination.
The first DeepSeek run is data. It is not yet a verdict. Until the retry, this episode reports what happened, not why.
What stays true regardless
Two things hold up no matter what the retry shows.
External review is non-negotiable. Same-session DeepSeek wouldn't have caught this. Same-tier DeepSeek in a different worktree wouldn't either. You need someone with no skin in the original output and the patience to compare the brief to the diff.
Self-reports that look like success without being success are the worst category of model output. Gemini's was a clean numbered list of fixes. DeepSeek's was a clean numbered list of fixes. Both contained false claims. Neither was caught by the model itself. The diff and the brief are the only sources of truth.
Where the journey is now
- Episode 1: Opus 4.7 max and GPT-5.5 max tied 5/5 on tight. Boring.
- Episode 2: Gemini 1/5 in 6 minutes — speed without verification.
- Episode 3 (this one, provisional): DeepSeek 0/5 with full confidence — wrong brief executed thoroughly.
Next episodes land when the DeepSeek retry runs with adjusted harness + sampler, when the workhorse-tier data lands (GPT 5.4 high, 5.4 medium, 5.5 low), or when the wide-scope evals collapse different conclusions.
Each episode is dated, frozen, and contributes evidence to a question the benchmarks alone don't decide. The slot is decided elsewhere — but each of these data points narrows the question of which models can be trusted with the bounded tasks that surround the slot.
The retry note isn't humility. It's discipline. A single run isn't a verdict. Two runs with different configurations and the same shape would be.