We went looking for a faster model. The speedup was already in the checkpoint.

For three consecutive research cycles we scanned the open-weight frontier looking for a better model to serve on our GB10 nodes, and each time concluded the same thing: nothing new in the 20 to 120B class, stay where we are.

That conclusion was correct and completely beside the point. The speedup we wanted was already sitting on disk, in the checkpoint we had been serving for months, behind a flag we never set.

Here is how we missed it, what it was worth, and two follow-up measurements that came out the opposite of what we predicted.

The physics that makes this matter

Decode on GB10 is memory-bandwidth-bound, not compute-bound. To emit a single token you stream the entire set of active weights out of memory.

273 GB/s  ÷  ~3.6 GB active weights (FP8)  ≈  76 tok/s

That is not a coincidence. It is almost exactly the 52 to 78 tok/s band we had measured for months. The compute units sit largely idle during decode, waiting on memory. You pay a full weight read to produce one token.

Anything that gets more tokens out of a single weight read is therefore worth more than anything that makes the arithmetic faster.

The thing we already had

Multi-token prediction (MTP) is exactly that lever. A small draft head proposes k tokens cheaply, then the full model verifies all k+1 positions in one forward pass, because verification is parallel rather than sequential. Verifying k tokens costs roughly what generating one costs, since both are a single pass over the weights.

It is also lossless. A draft token is accepted only if it matches what the target model would have sampled anyway, so the output distribution is unchanged. That is unusual: most speedups on this hardware cost you something (a smaller model, coarser quantization, less reasoning). This one costs nothing.

We had assumed our checkpoint did not have an MTP head. It did:

The drafter weights had been downloaded and idle on every node in the fleet since the day we deployed the model.

Why we missed it, which is the actually useful part

We had checked. We grepped the config for MTP keys, found none, and wrote “no MTP” into our decision log.

We grepped for num_nextn_predict_layers and nextn.

Those are DeepSeek’s key names. Qwen uses mtp_num_hidden_layers. Same capability, same runtime support, different vendor’s spelling, and our check was written against the wrong vocabulary.

The failure was not that the information was hidden. It was that we validated a negative against one vendor’s naming convention and then treated it as a property of the model. The config said what we asked it, and we asked it the wrong question.

The generalizable fix: confirm a missing capability against the artifact, not against the metadata. The tensor index is the ground truth. A weights manifest cannot misname 1,560 tensors into invisibility, but a config-key grep can miss all of them at once.

What it was worth

Same checkpoint, same container, same quantization, same flags, on one idle node. The only delta was the speculative-decoding config. Five prompts, median of three runs, temperature 0.

Acceptance ran between 55% and 95% depending on the prompt, with a mean accepted length of 2.5 to 3.8 out of a maximum of 4 at k=3. Tool calling continued to dispatch correctly, which was the risk we cared most about.

No new hardware. No container upgrade. No quality trade.

Backwards result 1: the gain grows with concurrency

We predicted the opposite, and said so in writing before measuring.

The reasoning seemed sound: continuous batching already amortizes the weight read across concurrent sequences, so at high concurrency there should be no idle compute left for speculation to exploit, and the wasted work from rejected drafts should start competing with real batched work. On x86 datacenter parts this is roughly what people report.

The advantage roughly triples between one stream and sixteen.

The explanation is that GB10 is bandwidth-starved enough that 24 concurrent sequences do not exhaust the idle compute. Batching amortizes the weight read across sequences; MTP amortizes it across tokens within a sequence. On this hardware those compose rather than compete.

The transferable lesson is narrower than “speculative decoding scales.” It is: do not import throughput intuitions from datacenter GPUs to a bandwidth-limited part. The ratio of memory bandwidth to compute is different enough that the tradeoffs invert. Measure it on the hardware you actually have.

Backwards result 2: output equality is the wrong correctness gate

MTP is lossless, so our first correctness check compared outputs before and after, hashed, at temperature 0. They differed. That looks like a serious problem.

It was not. Before concluding anything we ran the control: the same container, the same prompt, temperature 0, three consecutive times.

A single server disagrees with itself at temperature 0. With chunked prefill and prefix caching enabled, reduction order varies, one token flips, and the continuation diverges from there. Temperature 0 means greedy sampling, not bitwise reproducibility.

So byte equality could never have been a valid gate, with or without speculative decoding. The right check is answer correctness, and every run on both sides produced the right answer.

If we had skipped the control we would have “discovered” that MTP breaks losslessness, and filed a bug against a system that was working correctly.

The caveat that matters more than the speedup

Shortly after deploying this we got a report that a session felt slow. It did. The cause was not MTP.

The session in question had grown past 90,000 tokens. Production traffic logs showed the same curve, and crossed the deployment boundary in both directions, which ruled out the change we had just made.

MTP accelerates decode arithmetic. At long context the bottleneck moves to prefill on the uncached portion of each turn plus decode attention over an enormous KV cache, where every generated token attends across all 90,000 entries. Speculation does not touch either.

The model advertises a 262,144-token context, and that number is real and native, not a stretched rope-scaling figure. But the supported context and the usable context are different numbers. Below roughly 20k you keep 33+ tok/s. Past 50k the added history generally costs more in latency than it returns, and long-context reasoning quality degrades on the same axis for unrelated reasons.

A 39% throughput win is worth having. It is also worth roughly nothing next to a session that has grown 100x longer than it needs to be. Trimming context made answers both faster and better, and cost nothing at all.

What we took from this

1. Validate a missing capability against the artifact, not the metadata. Our config grep was written in another vendor’s vocabulary and returned a confident wrong answer.
2. Run the control before believing a divergence. Our correctness gate was measuring nondeterminism, not correctness.
3. Do not port performance intuitions across hardware classes. The concurrency result inverted on a bandwidth-bound part.
4. Look at what you already have before going shopping. Three cycles of frontier research produced a correct “nothing better available” and completely missed a 39% gain sitting in the checkpoint on disk.

The last one is the one that stings, and it is the one most likely to generalize to your setup.