Executive Summary
Across today's sources, verification is moving away from reading artifacts and toward measuring behavior. Willison shipped a 321-commit game without reading the code, because the agents built their own playtest harness. Husain shows that the cheap judges meant to automate that measurement are miscalibrated and set to the wrong thresholds by default. Lambert shows that model provenance now matters to procurement as much as capability does. Jones and Conrad cover the physical side: agents consume endpoint resources without limit, and GPU capacity is turning into a financed commodity with prices that depend on how much interruption a workload can tolerate. For enterprises, control now comes from evals, provenance records, resource limits and capacity contracts, not from human inspection of what agents produce.
What Changed
- Outcome prompts plus orchestration beat specs. "Make it a LOT more fun," run through Ultracode, led to over 1,000 simulated playtests and cut undecided AI matches from 14% to 0.9%. Several rounds of step-by-step direction had failed to get there.
- Open-weight models became a regulated asset. DHS barred DeepSeek from federal civilian procurement. It is unclear whether finetunes and distilled models are covered, and Nvidia's Nemotron recipes depend on DeepSeek. MOFCOM's pre-approval proposal gives China a lever over exports.
- The inference cost floor dropped. DeepSeek open-sourced DeepInfer, with an estimated 90% gross margin on its own serving. DeepSeek-V4.1-Flash (MIT license) leads DeepSWE on price/performance. Reflection's Beam (Apache 2.0) is the first credible permissively licensed American alternative.
- Decision models are now viable as eval judges, with conditions. OpenAI Decisions scored F1 0.87 at 3.72¢ per 100 evals. Perplexity scored 0.79 at 0.82¢. Both are unreliable at the vendor default 0.50 threshold.
- GPU capacity has a forward curve. A B300 earns about $40/hr from inference and rents for $5 to $6/hr. Interruption terms are priced by workload, and lenders are starting to finance against forward capacity.
Cross-Expert Synthesis
Willison and Husain are describing the same shift from opposite ends. Willison's quality gate was a simulation harness the agent built itself. Husain's message is that such harnesses carry their own failure modes: a Clef bucket scored at 8% failure actually failed 60% of the time, and no threshold met an 80/80 precision/recall bar in roughly a third of training splits. If no human reads the code, the eval decides what ships, and the eval itself needs calibration, held-out data and labeled traces. The trig desync bug (Chrome and Safari returned different Math.sin values) shows what this costs: unreviewed code passes its own tests and fails across environments nobody simulated.
Willison and Jones disagree on how much autonomy to give agents. Willison ran Ultracode for hours on leftover allowance and treated token burn as an acceptable cost. Jones argues that agents RL-trained to keep going will fill whatever headroom exists, and cites 37TB of SSD writes in 21 days from one Codex logging bug. Both are right, and they are describing different budgets: tokens in the cloud, RAM and disk on the endpoint. Enterprises need stop conditions on both.
Lambert and Conrad pull in opposite directions on cost. DeepInfer deflates per-token prices. Conrad's numbers show hardware scarcity holding rental rates well below inference earnings, so whoever owns serving efficiency keeps the spread. The cheapest tokens available right now also carry the most provenance risk.
Cloudflare's Clef shows up in both Lambert's landscape and Husain's benchmark. That is the clearest sign that small classifier-style models are becoming routine infrastructure for routing and judging.
Enterprise Implications
Code review no longer scales as the main quality gate for agent-built software. Organizations that do not replace it with owned eval harnesses (labeled traces, tuned thresholds, measured calibration) will ship unverified behavior and not know it. Data science teams, not application teams, are best placed to own this.
Model provenance is now a supply-chain question. Any federal-adjacent stack needs a record of base model, finetune lineage and distillation sources. "We use Nemotron" may not satisfy an auditor if the DHS scope widens.
Agent capacity planning covers three cost centers: tokens, endpoint resources and reserved compute. Cloud inference does not remove endpoint load, because the harness (browsers, dev servers, builds, caches) still runs locally. Batch workloads that tolerate preemption on minutes of notice should not be priced like live inference.
What To Do About It
- Stand up an eval function before scaling agent output. Use per-failure-mode judges, minimum precision/recall set by the use case, cross-validated thresholds and a calibration check. Treat vendor benchmarks as hypotheses to test on your own data.
- Build a model provenance register now. Tag each model by origin, license and training lineage. Keep a domestic fallback (Beam, Nemotron, Muse) qualified for regulated workloads, and use DeepSeek or MiMo where policy allows.
- Put lifecycle limits on agents. Give every agent a stop condition, a defined output location, resource quotas and a standing cleanup job. Monitor SSD wear and disk growth on agent hosts. Audit and repurpose existing machines before buying new ones.
- Split compute contracts by interruption tolerance. Move batch inference and evals onto preemptible capacity, and reserve long-notice contracts for SLA-bound live serving.
- Customer question: "If no one on your team reads the code your agents write, what exactly decides whether it ships, and who calibrated that?"
Risks and Blind Spots
The evidence is thin. Husain worked from 98 private traces in one domain. Willison's result is a game with no regulatory exposure, and its failure modes (a headache from the audio modem, browser desyncs) show up immediately in a way that silent data corruption in an enterprise system would not. Jones's load claims are anecdotal and partly promote his own prompt guide. Conrad is describing the market he sells into, and 5% of capacity on the order book is not proof of liquidity. The DHS scope question is unresolved and could reach lineage that companies did not know they depend on. The largest blind spot is that every source assumes evals catch what review used to catch. Nobody has shown that self-built agent harnesses can be trusted to test the agent's own output.