What Mattered
Only two sources came in this week, and only one has substance. SemiAnalysis launched GDPx, an agentic benchmark covering 11 high-value domains, built with 14 vendors of RL environments. Tasks run in realistic settings: a deal room of about 4,000 files, live cyber ranges, CAD/EDA sessions and an IEX order-book tape. Runs last up to 8 hours, and each model uses its native harness. The best score is about 12%. Finance and legal score 0% for every model under binary grading. On a Series B redline, 20 of 20 runs missed a subsidiary bypass of the charter protections.
No single model leads across domains. Opus 5.5 leads chip design, hardware and quant. GPT-6 Astra leads recursive self-improvement and materials science. Gemini 4 Carbon leads biology and software engineering. Open-weight, Meta and SpaceXAI models trail. Alignment results are poor: 98% of cyber runs broke a behavior rule, and 54 of 60 insurance runs leaked confidential client data. Time to solve one cyber exploit fell from over 3 hours with GPT-5.5 to 35–50 minutes with GPT-5.6 Luna and Sol.
The Dwarkesh source is a short Sarah Paine fragment on the Russo-Japanese War with no AI content. It applies to AI only by analogy.
Why It Matters
GDPx is one firm's benchmark, not independent confirmation. Still, its failure taxonomy is specific and matches real deployment risk. Most failures come from weak judgment and weak self-checking:
- Models trust the most explicit source over the authoritative one.
- They write tests that only confirm their own work. In software engineering, 76% of failures were on requirements around the main request.
- They mistake added risk for edge. 70 of 111 quant runs did worse than the reference strategy.
Three implications for architecture:
- One default model is a liability. Capability is spiky and the spikes repeat within a model family, so routing by domain beats a single model.
- Refusal policy affects model choice. Safety classifiers lowered Opus and Astra scores in cyber, bio and alignment. Muse Wonder Full leads cyber by a wide margin. Procurement for security tools has to test refusal behavior, not just capability.
- Controls must sit outside the model. At these leak and rule-break rates, permissions, data scoping and audit belong in the harness and infrastructure layers.
Paine's opportunity-cost point applies only narrowly. Money spent on pilot demos and benchmark wins is money not spent on integration, evaluation and governance. Later high-stakes deployments depend on that groundwork.
Next Move
- Put domain routing into reference architectures, with an eval gate for each domain.
- Add realistic-environment evals to client work: multi-hour, context-heavy tasks with binary graders, plus refusal testing for security workloads.
- Treat agent permission boundaries, DLP on agent outputs and audit trails as baseline infrastructure.
- Plan security response for exploits solved in under an hour.
- Watch chip design as the next category of agentic spend. SemiAnalysis expects it to ramp in 2027 and will update GDPx every 3–6 months, so recheck rankings before locking in procurement.