The methodology file went into the repository before the sweep did. It had to; that is the rule. Next to the cost model and the hardware description in efficiency-methodology.json there is a sentence written by people who had not yet measured anything: expect roughly an order of magnitude advantage in index points per gigabyte for a small specialist against a larger general model.
Then the sweep ran. The best ratio across four candidates was 4.12×. The worst was 2.16×.
The advantage was real. The framing was not. And the reason was not that the models underperformed. We had quietly picked the wrong opponent, the same one almost everyone picks when they argue for a small model.
The model your small model competes against is not the frontier, and it is not the 8B you keep in the comparison table because it makes the ratio look good. It is the best small general model a cost-conscious buyer would reach for instead. That baseline decides whether your efficiency story survives contact with a buyer, and it is nearly always harder than the one in the deck.
Who was actually winning the peer set
Our specialised axis is intake-bench: can a model turn a messy, sometimes hostile transcript into exactly the six-key JSON a downstream system needs, without being steered off it. When we measured general models on that axis — one harness, one prompt, greedy decoding — the best-scoring peer was not Ministral-8B and not Llama-3.1-8B. It was gemma-2-2b, at 2.6 billion parameters, tied with Ministral at 34.03 on the public split.
Against the 8B class, our fused 1.5B specialist carries roughly six times the points per gigabyte. Against gemma it carries between two and four. Both numbers are true. Only one describes the decision a buyer faces, because nobody deploying a 1.5B model into a memory-bounded environment was going to reach for an 8B as the alternative. They were going to reach for the best 2B they could find.
The order-of-magnitude framing assumed the competition was large. It was not. So 2.16×–4.12× went into the findings file with the declared expectation printed above it and the words “not met” beside it, and the threshold it was judged against — 3× on each of three ratios, also declared before the run — stayed where it was. Three of the four unfused candidates failed it. That is the criterion working, not a reason to revise the criterion.
We keep the embarrassing number for a plain reason: a buyer who re-runs the comparison will find it anyway, and the only question is whether they find it in our document or in their own.
The four columns you have to read together
A parameter count on its own tells you almost nothing about what a model will cost you to run. It has to be read next to peak memory, tokens per second on the hardware you actually own, and the cost of a thousand tasks once you fold the two together. On our Mac Studio, cleanly measured with one process on the GPU at a time, the two Qwen2.5 bases we build on look like this.
| Model | Params | Peak GB | tok/s |
|---|---|---|---|
| Qwen2.5-1.5B-Instruct | 1.5B | 2.97 | 126.92 |
| Qwen2.5-3B-Instruct | 3B | 5.82 | 73.46 |
Memory tracks the parameter ratio almost exactly, which is why points per gigabyte is the dimension a small model can never buy its way out of: a model twice the size takes twice the memory whatever you do to it. Throughput roughly tracks it too, at least on a workstation with no batching tricks. The interesting column is the fourth one, because cost per task is where specialisation earns its keep. A fused 1.5B intake specialist completes a thousand intake tasks for about $0.14 on our cost model; the fused 3B for about $0.20. Those two numbers are only comparable because they were produced the same way on the same box, and they say nothing about what the same model costs on a cloud GPU. Cost per task on local hardware is a dimension the public indices do not publish at all. The largest one reports cost per task for API models only, and excludes locally-run and quantized models from its tables.
The adapter tax, and the number we got wrong first
There is a fifth thing to check, and it bit us. A LoRA adapter served unfused — the base weights plus a separate low-rank delta applied at inference — runs markedly slower than the base it wraps, at essentially the same memory. Cleanly measured, an unfused adapter on the 1.5B ran at 65.69 tokens per second, 52% of its base. On the 3B, 61%. Nothing in a parameter count predicts that. If a deployment cares about throughput, the adapter has to be fused into the base before serving, and the fused artifact has to be re-measured rather than assumed to inherit either the base’s speed or the adapter’s score.
We did re-measure it. Fused, the 1.5B returned to 120.33 tokens per second, 95% of base, and its intake-bench score moved from 82.10 to 82.10 on the full suite. The 3B fused went from 85.80 to 85.51. Movements within a point, consistent with merging weights in bf16. Fusing buys throughput without buying a different product.
But that is the second version of the story. The first version was wrong, and how it was wrong matters.
Our first sweep reported the 1.5B base at 79.4 tokens per second and the unfused adapter at 25.8–26.6, a penalty of roughly 3×. On those numbers a fused model appeared to run 1.5× faster than its own base, which we wrote down and briefly believed. Then we found that two background chains had been waiting on the same process check and released at the same instant, so the base and peer sweep had run concurrently with a recalibration job. Only the fused measurements had run alone. The base had been understated by a third. Two claims were withdrawn in writing, the contended files were kept under an INVALIDATED.json marker because the discrepancy is itself evidence, and every efficiency artifact now records which GPU processes were present before and after the run, with a clean flag the index refuses rows without.
The lesson for anyone choosing a model from a table: the throughput column you did not measure yourself was produced under conditions you cannot see.
“Against the 8B it is six times. Why not say that?”
Because the 8B is not what you would have deployed instead. The honest comparison is the one a buyer can re-run and agree with, and a buyer choosing a 1.5B for a memory-bounded box compares it to the strongest 2B or 3B they can find. If we can beat that by 4–5.5× on cost per task, the claim survives. If we can only beat an 8B by 6×, the claim is decoration.
There is a sharper version of the objection: once fused, the speed and cost ratios improved to 5.52×, so surely the efficiency story is stronger than the memory ratio suggests. It is. But the expectation we declared named one dimension — points per gigabyte — and claiming vindication on a different dimension after the fact is the goalpost-moving the whole programme exists to prevent. The prediction fails on all three dimensions. We say so, and we say which dimension every time we quote a ratio.
What a defensible efficiency claim looks like, then, is a sentence in which every clause is a constraint: 4–5.5× more capability per unit of cost than the best-scoring small general model we measured, on our own axis and our own hardware, with the model fused. Remove any clause and it stops being true.
The baseline is going to get harder. The next measurement wave targets the families that now define the small class — Qwen3.5 at 0.8B, 2B and 4B, Gemma 4 E2B and E4B, SmolLM3-3B, Granite 4.1 3B — and we have not measured any of them yet. Until we have, none of them appears in a ratio, and the day one of them beats our specialist on its own axis, that number goes on the Field Index in the same row as everything else.