Every local-inference writeup leads with tokens per second. It is the number on the vendor slide, the number in the forum post, and the number people compare when choosing what to run on their own hardware.
It is also close to useless on its own, and we have the measurements to show why.
We evaluated two models on one GB10 node (128 GB unified memory, SM121), same stock NVIDIA NGC vLLM container, same serve flags, same quantization class. One was Qwen3.6-35B-A3B, our current daily driver. The other was Ornith-1.5-35B-A3B, a continued-pretrain of that exact base, which we confirmed by diffing the configs: identical 40 layers, 256 experts with 8 active, 2048 hidden, same 248320 vocabulary, same GatedDeltaNet split of 30 linear and 10 full attention layers.
Two models, same architecture, same hardware, same flags. A clean comparison.
Finding 1: on tokens per second, it is not close
| Ornith-1.5-35B-A3B | Qwen3.6-35B-A3B | |
|---|---|---|
| Single-stream tok/s (median of 3) | 39.4 | 54.7 |
Qwen is 39% faster per token. On the number everyone quotes, the decision looks made.
Finding 2: the slower model finished first
We then measured time to a finished answer across six prompts spanning a trivial arithmetic question through an open-ended agentic planning task, three runs each, medians reported.
| Prompt | Ornith | Qwen |
|---|---|---|
| short (17 x 23) | 89 tok / 2.3 s | 245 tok / 4.5 s |
| factual recall | 41 tok / 1.1 s | 163 tok / 3.0 s |
| multi-step reasoning | 400 tok / 10.2 s | 908 tok / 16.6 s |
| write a function | 272 tok / 6.9 s | 1366 tok / 25.0 s |
| open-ended explanation | 1335 tok / 33.8 s | 2065 tok / 37.3 s |
| agentic planning | 3188 tok / 81.1 s | 3992 tok / 73.3 s |
| total | 5325 tok / 135.3 s | 8739 tok / 159.7 s |
The slower model finished the suite 18% sooner, because it emitted 1.64x fewer tokens to say the same things. Correctness held: both got the arithmetic, the capital city, the reasoning problem, and a merge function that passed the same six edge cases when we actually executed the output.
The single most striking row is the first one. Qwen spent 245 tokens and 4.5 seconds to multiply 17 by 23. With reasoning enabled, that overhead applies to every trivial turn, and trivial turns are most of what a coding agent does.
Finding 3: the break-even is a ratio, and you can compute it
Our first instinct was a length threshold: “below N output tokens the faster model wins, above it the terser one does.” That was wrong, and the data says so. Look at the verbosity ratio per row: 2.75x on the short prompt, 5.02x on code, 1.55x on explanation, 1.25x on the longest agentic task, which is the one row Qwen won.
The relationship is not to length. It is:
speed_ratio = 54.7 / 39.4 = 1.39
verbosity_ratio = qwen_tokens / ornith_tokens
The terser model is faster to a finished answer only when its verbosity advantage exceeds the speed deficit. Here that means a verbosity ratio above 1.39. It clears that comfortably on short and structured work and falls below it on long open-ended generation, which is exactly where the ratio compresses.
That is a number you can compute for any pair of models on your own workload, and it is a far better selection criterion than either raw tok/s or a length heuristic.
Finding 4: four ways the measurement lies to you
These cost us real time, and every one produced a confidently wrong result.
A token budget that is too small looks like a broken model. Our first harness gave every prompt 64 tokens. Reasoning-enabled models spend that entirely inside an unclosed thinking block, so both content and reasoning_content come back empty. We briefly believed a production model was returning nothing. It was returning a truncated thought.
Truncation turns a time comparison into a throughput comparison. Two rows in an early run hit a 2048-token ceiling. A capped response does not measure time to an answer, it measures the cap. Raise the limit until both models reach finish_reason: stop, or discard the row.
A wrong answer key blames the model. We marked both models FAIL on a word problem because our checker looked for the wrong number. Both had been right. Any automated grader should print the model’s actual output on failure, so a broken checker is distinguishable at a glance from a broken model.
Background load quietly changes the verdict. One run overlapped with other traffic on the same node. That single contended row dropped the model from 39.3 to 28.9 tok/s and moved the aggregate result from +18% to +6%. Benchmark against an idle node, and check that it is idle rather than assuming.
The general defense against all four: run a known-good control alongside the model under test. Three of these four errors were caught only because the control failed in a way we knew was impossible.
Where that leaves things
Tokens per second measures the engine. What you care about is the trip.
For interactive and agentic work, the metric that matters is time from prompt to finished answer, and that is a product of generation speed and how much the model chooses to say. Post-training changes the second factor enormously, even between models that share an architecture byte for byte, as these two do.
Practically: compute the speed ratio between your candidates, then measure verbosity on prompts that resemble your actual workload. If the terser model beats the speed ratio, it wins on wall clock despite losing the benchmark everyone quotes.
One caveat worth stating. The longest agentic row was the noisiest across runs, with the verbosity ratio swinging from 1.85x to 1.25x and flipping the winner. Three runs was not enough to settle it. Treat single-run comparisons on long open-ended generation with suspicion, including ours.