hotic's blog

Why Inference Speed Is the Next Battleground for Coding Agents

TechJune 18, 2026

When we talk to a model, the wait before the first token is prefill; streaming tokens out one by one after that is decode.

On the decode side, Artificial Analysis tracked the output speeds of leading coding models over the past two years—from Claude 3.5 Sonnet to Claude Sonnet 4.6—and nearly all of them sit below 200 tok/s.⁠[22] Two outliers showed up in 2026: Codex-Spark in February, and GPT-5.6 Sol, which OpenAI previewed for wafer-scale deployment. Both run on Cerebras.⁠[4]

I think inference speed has quietly become the second axis of competition in AI coding. Models keep climbing the intelligence index, but when you’re sitting in an IDE going back and forth with a model on code, the cost of waiting becomes harder and harder to ignore.

Decode Is the Bottleneck, and It’s Memory-Bound

Generating a response happens in two phases. Prefill ingests the entire prompt in one pass, saturating tensor compute—it’s compute-bound. Decode streams tokens out one at a time, reading the model weights from GPU memory on every single token. That eats memory bandwidth (typically HBM bandwidth in a data center), making it strictly memory-bound.

Of course, not all the waiting you feel while coding comes from decode. Waiting on a JVM build, running Rust or Cgo tests, searching the codebase, or watching the main thread sleep—those tool calls take real wall-clock time too. That kind of latency is outside the scope of this post: you either hide it by running parallel sessions or throw better local hardware at it, and it has little to do with how fast the model itself decodes. It does point to a deeper question—a model’s effective “step horizon,” or how reliably it can get things right within a given window of time—which touches on deeper emergent capabilities.

Back to the model’s own output speed. A single GPU only has so much memory bandwidth, so without extra tricks, single-user decode on a frontier model usually lands in the tens of tok/s. There are only a few ways to speed that up: move the weights into memory that’s much faster than HBM; emit multiple tokens per weight read; or shard the model across many chips so their combined interconnect and memory bandwidth adds up. Push that logic to the extreme, and you etch the weights straight into silicon. Vendors today roughly fall into four camps:

Route Players Approach Trade-off
On-chip SRAM Cerebras, Groq Keep weights in on-chip SRAM, bypassing HBM Expensive hardware, tight capacity
GPU + speculative decoding Zhipu, Xiaomi, Fireworks Standard GPU nodes with speculative decoding and quantization Speed fluctuates with content
Pay for parallelism Anthropic Opus Fast Massive tensor parallelism on dedicated lanes High per-token price
Etched into silicon Taalas Hardwire weights into metal layers on the chip One chip only runs one model

Route 1: Move the Weights into On-Chip SRAM

Cerebras’s WSE line turns an entire 12-inch silicon wafer into a single chip, packing 44 GB of on-chip SRAM onto the WSE-3.⁠[5] When the weights live in SRAM, decode never has to touch HBM.

Right now, the trend looks like this:

  • October 2025: Cognition ships SWE-1.5 on WSE-3 at 950 tok/s.⁠[1]
  • February 2026: OpenAI launches GPT-5.3-Codex-Spark on dedicated WSE-3 hardware, rated at 1000+ tok/s—15× faster than GPT-5.3-Codex.⁠[2]
  • June 2026: OpenAI previews its flagship GPT-5.6 Sol and announces plans to bring Sol onto Cerebras in July at up to 750 tok/s for a small set of initial customers.⁠[3]

The first two were speed-specialized models; Sol is the flagship itself. Putting a flagship on wafers means the old assumption—that you have to drop down to a small model if you want speed—is starting to crack.

When a model doesn’t fit on one wafer, you tile it across several. In May, Cerebras ran the 1T-parameter Kimi K2.6 with 4-bit weights and 16-bit compute across roughly 20 CS-3 systems, routing inter-layer traffic over the wafer-scale fabric. Artificial Analysis measured it at 981 tok/s, 6.7× faster than the quickest GPU cloud.⁠[6]⁠[11] Community estimates suggest Sol uses a similar setup across 70 to 100 wafers.⁠11.The multi-wafer layout for Sol comes from community estimates and has not been confirmed by OpenAI. Placing one layer per wafer and streaming activations across wafers only adds to TTFT as depth grows, without slowing down per-token decode.

OpenAI has bet heavily on this line—extending a $1B loan to Cerebras in December 2025, followed by a $20B, 750 MW compute purchase agreement in January 2026.⁠22.Sam Altman, Greg Brockman, Ilya Sutskever, and Adam D’Angelo were early personal angel investors in Cerebras, per the Cerebras S-1 filing. Cerebras completed its $5.5B IPO in May 2026.

Groq’s LPU takes the same idea in a different direction. Each chip only carries 230 to 500 MB of SRAM at 80 TB/s of bandwidth, so it takes racks of hundreds of chips to hold a full model. Paired with speculative decoding, it runs Llama 3.3 70B at a measured 1665 tok/s.⁠[7]

A common question is whether this kind of hardware trades away accuracy for speed. An independent OpenBench run compared Kimi K2 on Together and Groq across MMLU categories, and every sub-score landed within a single percentage point.

Route 2: Speculative Decoding on Standard GPUs

Zhipu and Xiaomi take the software route, both explicitly running on standard 8-GPU nodes. GLM-5.1 Highspeed is rated at 400 tok/s,⁠[8] while Xiaomi’s MiMo-V2.5-Pro UltraSpeed—a 1T-parameter MoE—claims 1000+ tok/s.⁠[9]

Both sit on top of TileRT, a persistent-kernel runtime that keeps GPU SMs resident as a tile-level scheduling engine. The speedup mostly comes from speculative decoding: GLM trains Multi-Token Prediction (MTP) heads to predict several tokens per step, while MiMo uses DFlash for block-parallel drafting, accepting 6 to 7 tokens per round on average, alongside FP4 quantization on its MoE expert layers.

Cursor’s Composer Fast sits in this camp as well, hosted on Fireworks.⁠[10] Using eight B200 GPUs, NVFP4 quantization, and EAGLE-3 speculative decoding, Fireworks runs Kimi K2.6 at 348 tok/s—the fastest among public GPU API providers.⁠[11]

On the exact same Kimi K2.6, Fireworks hits 348 tok/s and Cerebras hits 981 tok/s—nearly a 3× gap. But Fireworks runs on GPUs you can actually buy off the shelf.

Route 3: Pay for Parallelism

Anthropic hasn’t disclosed the hardware behind Opus Fast. It doesn’t use a draft model; the main model computes every step. The speed comes from sharding the model across much wider tensor parallelism so the aggregate interconnect and memory bandwidth can feed decode, paired with dedicated capacity lanes for Fast traffic.⁠33.Anthropic has not disclosed the underlying hardware for Opus Fast; this section synthesizes official documentation and industry estimates.

The tradeoff is right there on the price tag. Opus 4.8 Fast costs $10 per million input tokens and $50 per million output tokens.⁠[12] In the Cursor chart above, Opus 4.6 Fast also has the longest price bar by far.

Route 4: Etch the Model into Silicon

The first three routes still read weights out of some form of memory—they only differ in where the weights sit and how many tokens you get per read. This fourth one is different. It’s the unorthodox path: etching the weights straight into circuitry.

This blew up on X for a while. In February 2026, a Toronto startup called Taalas unveiled HC1, an 815 mm² chip on TSMC 6nm with the weights of Llama 3.1 8B etched across the entire die. Out of roughly a hundred mask layers, only two metal layers are customized per model, hardwiring the weights into silicon while keeping the KV cache and LoRA adapters in writable SRAM.⁠[23]⁠[24] Taalas rates it at 17,000 tok/s for a single user, compared with roughly 1,900 tok/s for the same model on Cerebras.⁠[25]

The catch is just as obvious:

  • One chip, one model: changing models means new masks and a new tape-out. Taalas says it takes two months from receiving a model to getting silicon back.
  • Aggressive quantization: HC1 mixes 3-bit and 6-bit parameters, and the company admits accuracy drops compared with running the same model on GPUs.
  • Small capacity: an entire 815 mm² die only fits an 8B model.

AMD announced its acquisition of Taalas in August, planning to fold the technology into the Instinct roadmap.⁠[26] The next-generation HC2 moves to standard MXFP4 at roughly 20B parameters per die, slated for late this year or early next.⁠[27]

Still, I don’t think this route is completely without merit—it just depends on where the ceiling for model capability sits and where diminishing returns on intelligence kick in, so the real question is when a model becomes worth etching. Models iterate fast right now, with Opus going from 4.6 to 5.5 within the year and benchmark scores climbing steadily. Between a two-month tape-out cycle and multi-year hardware depreciation, a model can be a generation behind the moment the chip comes back from the fab. Yet once intelligence crosses a certain threshold, it’s good enough for day-to-day coding in an IDE. With a model at the level of Opus 5.5, you rarely need anything stronger for routine code—and if freezing it into silicon buys you a 10× speedup, that trade starts to look compelling.

Scale and cost are what hold it back. Anthropic hasn’t disclosed Opus 5.5’s parameter count, but frontier flagships sit in the trillion-parameter range. At 20B parameters per HC2 die, a 1T model takes 50 chips, and the ~3T Kimi K3 from the wafer chart would take around 150—all of which have to be re-spun every time the model updates. Nor can you quantize as aggressively as HC1; if a flagship loses points in the process, etching it defeats the purpose. So in the near term, it’s much more realistic to etch small and mid-sized models—something on the scale of Qwen 27B—while frontier flagships wait until the intelligence curve flattens out or per-die capacity multiplies several times over.

Fast on Average vs. Consistently Fast

Most of the numbers above are averages or peaks. In practice, speculative decoding and hardware parallelism feel very different day to day.

Speculative decoding lives and dies by its acceptance rate. When the draft model guesses right, several tokens sail through at once; when it misses, the main model falls back to one token at a time. Boilerplate and predictable code fly by, while dense reasoning slows right back down. Of course, the community has come up with all kinds of clever tricks here, and the push to squeeze more out of hardware and algorithms hasn’t slowed down—evolving from standalone draft models to native MTP and EAGLE-3 heads, block-parallel drafters like DFlash, and DSpark’s parallel drafting paired with a lightweight Markov correction head.⁠[29] Even so, what they raise is still the average, not the underlying decode bandwidth ceiling.

Parallel hardware computes every step in full, so speed doesn’t care how hard the content is—the line stays flat. Cerebras falls into the same bucket, except a single wafer is already big enough that it doesn’t need to stitch dozens of GPU cards together.

If you want speed that is both high and steady, hardware is still the only way. Speculative decoding lifts the mean, not the floor.

What Speed Actually Buys You

When Cognition launched SWE-1.5, they wrote two lines that stuck with me:

Developers shouldn’t have to choose between an AI that thinks fast and one that thinks well.
With a model running at up to 950 tok/s, previously negligible system delays emerged as dominant bottlenecks.

SWE-1.5, Cognition

The first line says you can have both speed and quality. On SWE-Bench Pro, SWE-1.5 lands right next to Haiku 4.5 and a few points behind Sonnet 4.5, while running 7 to 13× faster.

The second line is the practical one. Once the model itself gets fast enough, tool calls, network round-trips, and compiler latency—things that used to be rounding errors—become the main bottleneck, forcing you to rethink the entire agent harness.

What speed is worth depends on how you work with the model. One mode is interactive co-creation: you sit in the IDE steering direction and taste back and forth, where every ten-second pause breaks your flow. The other is autonomous delegation: you hand the agent a goal and let it run for half an hour, where raw token speed matters far less than getting it right on the first try.

In interactive work, once a model is smart enough, the gap comes down to speed. You can see the current landscape in Artificial Analysis’s intelligence-versus-speed scatter plot: flagships like GPT-5.5 and Claude Opus 4.7 all cluster below 100 tok/s, while Gemini 3.5 Flash scores 55—just 5 points behind GPT-5.5—at roughly 280 tok/s, sitting alone in the upper-right corner.⁠[13]

For long autonomous runs or unfamiliar domains, paying the flagship intelligence premium is still worth it. Research-heavy work will keep defaulting to frontier models, and if you want speed there, you pay Opus Fast prices. If most of your day is interactive coding, the high-speed tiers are worth a serious look.

November 2026 Update

The main post above was written in June. Things move fast in AI, a lot has happened over the past few months, and parts of this story played out differently than I expected back then.

Wafer-Scale Sol Never Went Mainstream

On August 13, OpenAI and Cerebras announced the Ultrafast tier for GPT-5.6 Sol on WSE-3, rated at up to 750 tok/s.⁠[14] It has remained a limited API preview for a handful of customers like Jane Street and Podium, with no public pricing and no general availability to this day.

Cerebras rolled out a clock-boosted WSE-3T in August alongside the CS-4, which packs three wafers into a single cabinet. In October they tiled the ~3T-parameter Kimi K3 across 59 WSE-3T wafers to hit 1,000 to 2,000+ tok/s, shown in the second view of the wafer chart above. Still, given the cost, I doubt a setup like this is practical for everyday use—it reads more like a technical flex, something reserved for big enterprise contracts or specialized workloads.

At DevDay on September 29, OpenAI launched another Ultrafast tier—this time for GPT-6 Astra, which is a different beast altogether. While August’s blog post touted 750 tok/s for Ultrafast, by the time it actually launched, NVIDIA’s blog noted it runs on Blackwell GPUs at a rated ceiling of just 300 tok/s in Codex.⁠[17] In Artificial Analysis benchmarks, Astra Ultrafast clocks in around 188 tok/s, while GPT-6.1 Sol Ultrafast (launched in October) hits roughly 220 tok/s.⁠44.Starting in November, Artificial Analysis switched the primary metric on its speed leaderboard to end-to-end task completion time; the tok/s figures here are taken from the output speed column on the same page, which varies by roughly 10% across snapshots. Benchmarks are one thing, though, and live production traffic is another.⁠[28]

Ultrafast is priced at 6× the standard tier, putting Astra Ultrafast at $60 per million input tokens and $300 per million output tokens.⁠[15]⁠[16] Yet in real OpenRouter production traffic, openai/ultrafast averaged just 108 tok/s over the week. GPT-6.1 Sol Ultrafast fared even worse, averaging 61 tok/s for the week—barely distinguishable from the 2×-priced Fast tier (59 tok/s) or even the standard tier (59 tok/s). On the subscription side, Ultrafast is locked behind the $500/month Pro 500 plan plus Enterprise and Edu tiers, leaving individual developers on the outside looking in.

Opus Fast Quietly Became the Value Pick

Anthropic went the opposite way. Fast mode on Opus 4.6 and 4.7 used to cost 6× the standard rate, before dropping to 2× with Opus 4.8 in May. In September, Opus 5.5 cut prices by 20% across the board—bringing the standard tier to $4 input / $20 output per million tokens and Fast to $8 / $40—while graduating Fast from research preview to general availability.⁠[18]⁠[19] With Claude Fable 5.2, released on November 10, Anthropic brought Fast mode to the Fable line for the first time. Fable 5 and 5.1 never had a high-speed tier, so this move is almost certainly a direct response to GPT-6 Astra’s Ultrafast. Standard Fable 5.2 also dropped from $10 / $50 per million tokens down to $8 / $40, meaning Fast at 2× comes out to just $16 / $80 for up to 2.5× higher output speed.⁠[12]⁠[19] That’s roughly what standard Opus 4.1 cost ($15 / $75) when it launched a little over a year ago—except today that same price bracket buys Fable 5.2 at Fast speeds, with intelligence in a completely different league.

On the official endpoint, Opus 5.5 Fast averaged 154 tok/s over the week—roughly double the standard tier, and noticeably faster than Astra Ultrafast at a fraction of its $300 price tag. Both companies charge a premium for speed, but OpenAI’s 6× Ultrafast tier regularly sags under real load, with Sol Ultrafast dropping to 33–36 tok/s between November 5 and 8, slower than its own standard tier. Anthropic charges 2× for Fast and holds its speed far more consistently.

Subscriptions Keep Getting Slower

Meanwhile, ChatGPT subscribers have spent the past few months complaining about Codex feeling sluggish. Complaints peaked after GPT-6.1 Sol launched in late September, when a developer on X benchmarked OpenAI and Anthropic across both direct API and subscriptions using identical prompts: the same GPT-6.1 Sol ran at 73 tok/s over the API versus just 22 tok/s in a Codex subscription, and even Codex Fast (46 tok/s) trailed the standard API—whereas Opus 5.5 (97 tok/s) and Fable 5.1 (96 tok/s) in Claude Code subscriptions ran at nearly full API speed.

User-reported benchmarks under varying setups; only the order of magnitude matters. The last two posts are in Chinese, both complaining that Codex keeps getting slower. Source: X, LINUX DO.

WorkOS ran a controlled comparison in September using the same GPT-5.6 Sol model across identical agent tasks on API keys versus subscriptions:

  • Standard tier: 39.2 tok/s on the API vs. 28.1 tok/s on subscription—1.5× slower end-to-end on subscription, with no difference in output quality.⁠[20]
  • Subscription with Fast enabled: roughly matched the default API speed without Fast.⁠55.WorkOS ran 10 trials per group on a TypeScript agent benchmark, finding a median ratio of 1.02× (0.94× after excluding outliers).

On October 5, Tibo Sottiaux announced a ~50% default speed bump for subscription users, effectively confirming that subscription traffic had been throttled.

Source: X.

Flat-rate monthly subscriptions are essentially compute subsidies. When capacity gets tight, single-user throughput is traded away for higher concurrency—and because subscribers are the ones being subsidized, their experience is the first thing on the chopping block.

Source: X.

I suspect he’s the only one running the real, full-blooded Ultrafast—or that the true Cerebras wafer build is still being kept in-house.

Speed Comes with Steep Costs

Right now, the vendors that can still offer high speed for cheap roughly fall into two buckets. One is providers with spare inference capacity whose models aren’t as supply-constrained. The other cuts costs in the model architecture itself, with October’s DeepSeek V4.1 Flash 1017 as the prime example. Built on a Causal Encoder-Decoder (CED) architecture, its 552B-parameter MoE activates just 8B parameters on input and 16B during decode.⁠[21] Fewer active parameters mean fewer weights to read per token, making decode inherently fast. On its official OpenRouter endpoint, it costs just $0.60 per million output tokens at a 98 tok/s weekly P50 average, and switching to third-party providers like Together or Baseten Fast pushes that to 232 or even 580 tok/s. China’s domestic inference stack is moving in the same direction, with V4.1 Flash hitting 2,469 tok/s per card and 5 ms per token for a single user in Huawei’s offline Ascend 950 tests.⁠66.Huawei’s internal offline benchmark at EP32 and 128K context, excluding scheduling overhead. Whether DeepSeek’s live API runs on Ascend hardware has not been publicly disclosed. When DeepSeek launched V4, they promised steep price cuts once Ascend 950 supernodes ship in volume, though broad 950DT supply isn’t expected until late this year or early next.

So the way I see it now, inference speed is still a real competitive axis, but it comes with steep underlying costs. Prefill leans on chip compute; decode leans on memory bandwidth. Wafer-scale capacity from players like Cerebras is finite, while the GPU side is constrained by HBM3e and HBM4 supply, making it just as expensive to brute-force bandwidth through massive parallelism. As for Ultrafast, long may it live!

References

  1. [1]CognitionIntroducing SWE-1.5
  2. [2]OpenAIIntroducing GPT-5.3-Codex-Spark
  3. [3]OpenAIGPT-5.6
  4. [4]Artificial AnalysisGPT-5.6 Sol Providers
  5. [5]CerebrasWafer-Scale Engine
  6. [6]CerebrasKimi K2 Enterprise
  7. [7]GroqLPU Architecture
  8. [8]ZhipuGLM-5.1 Highspeed Documentation (in Chinese)
  9. [9]XiaomiMiMo × TileRT 1000 TPS (in Chinese)
  10. [10]CursorComposer 2
  11. [11]Artificial AnalysisKimi K2.6 Providers
  12. [12]AnthropicFast Mode Documentation
  13. [13]Artificial AnalysisGemini 3.5 Flash Evaluation
  14. [14]CerebrasAccelerating GPT-5.6 Sol Ultrafast with OpenAI
  15. [15]OpenAIUltrafast Mode Documentation
  16. [16]OpenAIAPI Pricing
  17. [17]NVIDIAGPT-6 Astra Ultrafast
  18. [18]AnthropicIntroducing Claude Opus 5.5
  19. [19]AnthropicPricing Documentation
  20. [20]WorkOSCodex allowance runs out? Use an API key, not credits
  21. [21]DeepSeekDeepSeek-V4.1-Flash Model Card
  22. [22]Artificial AnalysisLLM Leaderboards
  23. [23]TaalasThe path to ubiquitous AI
  24. [24]Unite.AIAMD Buys Taalas to Put Hard-Wired AI Models in Its Accelerator Roadmap
  25. [25]heiseAI inference cast in silicon: Taalas announces HC1 chip
  26. [26]AMDAMD Acquires Taalas
  27. [27]Tech TimesAMD Buys Taalas to Hardwire AI Models Into Silicon
  28. [28]OpenRouterEndpoint Throughput & Latency Benchmarks
  29. [29]NVIDIACo-Designing AI Models Using Speculative Decoding for Faster LLM Inference