The pattern
Two of the models in this week's batches independently shipped this:
use App\Models\Booking\Booking;
// ...
'status' => Rule::in(Booking::BOOKING_STATUSES),
Booking::BOOKING_STATUSES doesn't exist on App\Models\Booking\Booking. It exists on App\Entities\Booking — the Doctrine entity in the parallel namespace. The Eloquent model and the Doctrine entity have the same class name. They live two directories apart. They coexist because the project is mid-migration.
php -l doesn't catch it. The IDE doesn't catch it. The first PATCH against the endpoint hits Error: Undefined class constant BOOKING_STATUSES and 500s.
Gemini wrote one of these on F5. MiniMax 2.7 wrote three of these on F5 and F10. Each model wrote the same defect class independently. Neither flagged it as a risk. Both passed their own self-review.
The framing shift
I came into the benchmark thinking I was measuring models. Today the lens inverted.
The defect emerges where the two ORMs meet. It's the kind of bug that only exists because the project is in coexistence — both ORMs visible, similar names, similar shapes, different constant sets. A pristine Laravel project wouldn't have this trap. A pristine Symfony project wouldn't either. The trap is real because the seam exists.
Models don't get this wrong because they're bad. They get this wrong because the architecture is teaching them to get it wrong: the namespace looks right, the type looks right, the test for "is this thing actually defined" requires touching files in two different layers. Two models hit the trap. A third one running tomorrow probably will too.
What the followup said
When I asked MiniMax to explain, it did the grep I should have done before the eval: grep -rn "BOOKING_CANCEL_STATUSES" app/Models/ returns no results. grep -rn "BOOKING_CANCEL_STATUSES" app/Entities/ returns three. The model articulated the single-step verification: check app/Models/ versus app/Entities/ before writing the import. One grep would have caught all three instances.
This is the architectural mitigation: not "use better models," but add the lint rule that catches Eloquent-import + missing-constant before runtime. Static analysis. A PHPStan rule. A custom Duster step. The rule has to live at the architecture layer, not at the model-selection layer.
Because here's the routing implication: every model running this brief on this codebase is exposed to this trap. The defect won't be fixed by picking a smarter model. It'll be fixed by closing the seam where the trap lives.
What this means for the benchmark
The benchmark wasn't measuring just models. It was measuring the architecture's hostility to models — and the introspection layer was telling us where the architecture cuts. Two independent cuts on the same edge means the edge is sharp. Every other model that touches this seam will produce the same cut shape.
That's a different research question than "which model is best." It's "where is the system-under-test hostile to LLM agents, and what guardrails make hostility safe?" The benchmark generates both kinds of evidence, and they route to different actions:
- Model-quality evidence routes to model selection.
- Architecture-hostility evidence routes to architecture guardrails.
The Berceuse pipeline is going to need both. Routing decisions pick models that survive the trap; guardrail decisions add lint rules that close the trap so survival is cheaper.
I keep underestimating how much of this work is going to be about teaching the architecture to be safe for agents, not just about teaching the agents to be safe in the architecture.