State of AI with Nathan Benaich - Compute scarcity is an engineering problem

0:00Compute scarcity is an engineering problem. With Angeles Paravolaropolis, a research engineer at 11 labs at Rise 2026, there are not enough GPUs and no near-term fix. They're hard to find. And once you do, procurement can run for months before they serve traffic. Demand, meanwhile, climbs exponentially. Angeles Paravolaropolis built his Rise talk on that mismatch. And on the only honest response to it, if you cannot add hardware, you make the most of what you have.

0:29For the voice inference workload he walked through, his talk measured how far that goes, counted in users served per GPU, from 1 to 70 with standard engineering, and to 140 at the frontier. Angelos leads 11 labs speech-to-text and text-to-speech teams and built its Scribe and Scribe real-time transcription systems. The Scribe V2 models he shipped this past year rank, he says, as the most accurate transcription models on most popular benchmarks.

0:58It gives him a particular vantage on the problem, since voice models live or die on latency and cost at scale. It is also the second year running 11 Labs has taken the rise stage. In 2025, its CEO, Mati Stanisiewski, spoke on the voice frontier. This year, we dove into the engine room. What a token actually costs. Every optimization starts with knowing what you're paying for. For the autoregressive transformers behind most popular LLMs, A token's cost reduces to two bottlenecks. Compute, how fast the GPU does the matrix multiplications and memory bandwidth, how fast the GPU's VRAM can load the model's weights, and its KV cache. Generation runs in two phases. A pre-fill step reads the whole prompt and fills the KV cache, the model's working state, and is compute heavy. A decode step then emits tokens one at a time, each conditioned on the last, and is memory heavy.

1:58The KV cache is what lets the model reuse that pre-fill instead of recomputing it for every new token. And at scale, it is the thing that hurts. 100 concurrent requests need 100 separate caches resident in memory. Size is not destiny either. Angelus noted that Quen3's cache costs almost three times as much per token as Quen2.5's despite near identical parameter accounts. So two models of the same size can cost wildly different amounts to run. Stop letting the GPU sit idle.

2:29The first and biggest single win is batching. GPS are excellent at parallel work and poor at sequential work, and the dominant cost in decoding is loading the model weights, which can be shared across every request in a batch rather than reloaded for each. Naive batching groups requests once and then waits for the slowest one to finish while the GPU idles. Continuous batching fixes that. It batches at the level of each decode or prefill step.

2:56so a new request can join a GPU already mid-flight on others. That one change takes 11 labs from serving one customer per GPU to 15. Shrink the weights, then the cache. With the GPU busy, the constraint becomes memory, so the next moves all reduce it. Quantization comes first. Models are usually trained at BF16, 16 bits per weight, which is more precision than they need. Dropping the weights to FP8 roughly halves their footprint with near lossless accuracy.

3:28given H100 class hardware and a little quantizerware training, which injects noise into the gradient so the model learns to tolerate the lower precision. That buys headroom for more cache and lifts throughput to 20 users per GPU. The more aggressive options exist too. Int4 is lossy but useful on-device, and MXFP4 reaches 4 bits but only on Blackwell and newer. Speculative decoding comes next.

3:53A cheap draft model proposes tokens and the big model verifies them in a single forward pass, accepting the run until the two disagree. It only pays off when the models agree often, which they frequently do not, so in practice it is used less than its reputation suggests. Applied here it nudges the running total from 20 to 28 users per GPU. The more popular cousin is multi-token prediction, where the same model wears extra prediction heads and drafts several tokens itself.

4:23with no second model to host. It earns its keep with two or more heads, and it doubles as a training signal. Teaching a model to anticipate several moves ahead, like a chess player, tends to make it more stable and sometimes faster to learn. Most big labs use it, and it lands in the same place, around 28 users per GPU. The largest gain is also the riskiest. The KV cache holds far less redundant capacity than the weights do, so compressing it is genuinely lossy.

4:50Angeles was candid about this from his own testing. The much discussed Google method TurboQuant was announced as lossless, but in his experience proved lossy in practice because any change to the cache is hard for the model to recover from. The fix is again on the training side. Distill the model so it grows accustomed to a lower precision FP8 cache and you keep most of the accuracy while shrinking the cache 2.5 times.

5:17That single step lifts throughput from 28 to 70 users per GPU, 70 times what the same hardware served at the start, where the frontier labs go. 70 is what discipline engineering gets you. Going further means changing the architecture itself, and here the labs are placing different bets. DeepSeq's multi-head latent attention squeezes each token's key value pair into a small latent rather than storing it in full.

5:45which both speeds inference and stretches context toward a million tokens. It was one of the more copied ideas after DeepSeq R1. QuenSwap's standard quadratic attention for a linear network on every other layer, cheaper and longer context, at some cost to quality. NVIDIA goes furthest, replacing the transformer on a fraction of its layers with state space models that scale linearly and compute faster, keeping enough transformer layers to hold accuracy up.

6:14With architecture level changes like these, the ladder reaches roughly 140 users per GPU. Nothing here is free. Angelos was careful not to oversell any of it. Every technique on the ladder carries a cost. Batching adds latency and runs into a memory ceiling. FP8. Quantization takes a small quality hit without the extra training. Speculative decoding needs access to weights and a training pipeline to work well. KV cache compression is the one most likely to degrade output.

6:46So the real question is how much degradation you can absorb rather than whether you can avoid it. He was blunter still about the gap between papers and production. Many compression methods that report no loss of accuracy were tuned on a handful of benchmarks and scaled to millions of users they can simply fall apart. You often only find out which ones once they're popular enough to be stress tested in the wild. Which technique pays depends entirely on the workload.

7:12The reason any of this matters beyond the engineering reached the room through a question from the floor. Today's token prices are subsidized by one audience estimate a factor of 10 to 40. Angelos' hope is that optimization, not subsidy, eventually closes that gap. The largest models, he said, have to be subsidized to make economic sense. But he expects smaller, sonic class models to become good enough for nearly all everyday use at margins that actually work.

7:39He already sees the shape of it in agent systems, route each request to the smallest model that can handle it, and reserve the expensive one for planning. Eleven Labs, whose models are smaller by design, is betting on the same logic, squeeze the hardware hard enough, and the economics start to look less like subsidy and more like engineering.