Executive Summary
Every layer of the stack in today's sources shows the same pattern: the number or claim being advertised is not the thing a buyer actually gets. Leaderboard scores get gamed at the level of training data. Sparse attention cuts attention compute but still needs the same HBM capacity. Cost per token flips between vendors depending on the latency target. An agent told a buyer "I'm here" without any signal that it was true. Across all of it, the advantage goes to organizations that measure inside their own workload, at their own SLA, with their own verification gates. Separately, the market is splitting inference into decisional and generative workloads. That makes "which calls actually need an LLM" a live cost question, not a theoretical one.
What Changed
- Jev is a model built only for decisions. It returns one discrete choice and never generates text, so there are no output tokens to pay for. It is designed to be called as a subroutine from existing agent stacks. The adoption claims are superlative and unverified.
- Serving hardware has no single winner. InferenceX's Sept 28 snapshot: GB200 leads MI355X (ATOM) by about 5 to 12% on cost per token at 125 to 150 tok/s. ATOM leads by about 9 to 13% at lower throughput or under a 2 second TTFT cap. GB200's fastest configs show 14 to 19 second first-token waits against about 1.1 to 1.2 seconds on ATOM. GB300 is 26% cheaper only if you accept 10 second TTFT.
- Sparse attention has a capacity catch. Top-k selection still needs the full KV cache resident in HBM. Actual capacity relief comes from the serving engine, for example HiSparse tiering the cache out to host DRAM.
- Serving-engine tricks now rival architecture gains. Chunked prefill cut TTFT from 28.6s to 8.7s. CUDA graph consistency roughly halved TPOT. TileRT gave MI355X a 40% interactivity edge over GB300 NVL72.
- Chinese labs are designing models around non-Nvidia chips. GLM-5 uses half DeepSeek's query-head count, and Z.ai had day-0 support deals with Moore Threads.
Cross-Expert Synthesis
The TypeSafe CEO and SemiAnalysis reach the same conclusion from opposite ends of the stack. The CEO says public benchmarks are structurally compromised because labs build data pipelines shaped like the test distributions. SemiAnalysis shows that even honest hardware benchmarks mislead unless you pin them to a latency SLA. Put together: a headline number is only useful after you have fixed the workload and the constraint it was measured under. Neither source thinks a generic comparison survives contact with a specific deployment.
Willison's Muse incident is the same problem at runtime. The CEO wants reliability measured continuously against real tasks ("moving the nines"). The Muse agent had no mechanism to check its own claim before making it, and its proposed fix only patched one reply template. Eval harnesses catch drift over weeks. Verification gates stop a single false assertion before it reaches a customer. You need both, and most agent deployments have neither.
There is a real tension around Jev. The Latent Space episode, framed around Jev, argues against public benchmarking. The Jev pitch rests on adoption velocity and unit cost. A vendor that rejects benchmarks is also a vendor that avoids head-to-head comparison. The CEO's own advice (distrust vendor claims, including ours) is the right way to read that, and it applies to Jev first.
Enterprise Implications
Model selection and hardware procurement are becoming ongoing measurement programs rather than one-time purchases. Two consequences for enterprise architecture follow.
First, the unit of evaluation is now the whole system: model, serving engine, memory tiers, and latency SLA together. A sparse-attention model on a naive serving stack does not deliver the advertised savings. Long-context agentic serving has to budget host DRAM capacity and bandwidth, not just HBM. In SemiAnalysis's data, GPU-resident cache reuse fell from 90.3% to 54.8% as concurrency went from 8 to 16, while host-memory reuse rose from 6.0% to 40.3%.
Second, agent workloads should be split into decisional calls (routing, triage, approve/reject) and generative calls. Decisional calls running on general-purpose LLM endpoints are probably overpaying, and they are also the calls where a wrong answer can be scored exactly. That makes them the easiest place to start building evals.
For hardware concentration risk, Chinese labs co-designing against Moore Threads, plus MI355X winning latency-bound tiers, means a mixed fleet is now an economic option and not only a hedge.
What To Do About It
- Build a workflow eval harness before the next model or vendor decision. Take a few hundred real production tasks with known-good outcomes and rerun them on every model or serving change. Treat vendor benchmarks as marketing.
- Put a TTFT SLA on every workload (interactive, near-real-time, batch) before comparing cost per token. Batch agentic work may justify GB300. Interactive work may favor MI355X.
- Audit agent call graphs for decisional calls and test a narrow classifier against your current LLM on your own labeled data. Pricing is a secondary check.
- Add a pre-deployment rule for customer-facing agents: any claim about real-world state (presence, availability, confirmation, SLA) must trace to a system of record, or the agent has to say it is unverified. Scope apology and remediation authority explicitly.
- Ask customers this: "Which of your agent's outbound statements can it prove, and what happens when it can't?" Most teams have no answer, and that gap is where an engagement starts.
Risks and Blind Spots
Jev's economics as reported ("$42 per 1,000 tokens sent a million times") do not resolve to a clear unit price. Verify the pricing before quoting it to anyone. Its fastest-adoption claim is self-reported and days old. Classifiers are cheap until the label set drifts. Discrete outputs with no reasoning trace are harder to audit when a routing decision goes wrong, so pair any classifier offload with logged inputs and periodic human review.
InferenceX is a single snapshot. Serving-engine gains like TileRT and chunked prefill move rankings from month to month, so hardware conclusions go stale quickly. Build re-benchmarking into procurement instead of locking in multi-year assumptions.
Workflow-specific evals have their own failure mode. Harnesses built on today's task distribution will quietly overfit to it, which is the same gaming problem moved in-house. Refresh eval sets from live traffic and track how many cases get added over time.