The model landscape, priced and sourced.

Static suites (MMLU, GSM8K, HumanEval) are saturated or leaked into training data. Prefer contamination-guarded boards — LiveCodeBench collects problems after model cutoffs; LiveBench replaces ~1/6 of questions monthly.

Harness choice alone swings scores 10–20 points; HLE diverges 10+ points between tool-assisted and closed-book runs; the same Terminal-Bench name shows different leaders on different harnesses. Compare within one harness only.

Arena's top tier sits inside ~55 Elo, and top SWE-bench Verified scores sit within ~1 point — inside the confidence interval, ordering is noise. Treat the cluster as one tier.

The Leaderboard Illusion analysis (2M battles) documented undisclosed private variant testing and preferential sampling on arena-style boards; pairwise voting also rewards confident-wrong over hedged-correct answers.

Practitioner consensus: leaderboard position has little correlation with whether a model works on your task. The leaderboard is the shortlist; your own evaluation is the decision.

Output ≠ input

Output tokens list at 4–8× input across the majors, so a generation-heavy workload prices completely differently from a summarization one.

Prompt caching

Cached repeated input is discounted up to ~90% — a long, stable system prompt changes the math entirely once caching is on.

Agent loops multiply everything

A single coding-agent task consumes 400K–2M cumulative input tokens across its loop; retries and tool-call fan-out multiply requests again.

Token efficiency varies by model

Models differ several-fold in output tokens spent per task solved — one Aug 2026 comparison measured a 4× spread on the same agentic benchmark — so $/task, not $/token, is the number to compare.

Long-context surcharges

Several majors re-price above a context threshold (e.g. higher rates past ~200–272K input tokens) — long-document workloads should be priced at the surcharge tier.

Deployed demand follows price-performance

Gateway usage data shows production traffic concentrating in 'good enough and far cheaper' models rather than leaderboard leaders — benchmark rank and deployed share are nearly uncorrelated.

Pretraining from scratch

Learning a base model from raw corpora — the frontier-lab lane, now at ~10²⁵–10²⁶ FLOP per run.

When: Almost never the enterprise answer: the EU AI Act attaches systemic-risk duties above 10²⁵ FLOP, and open bases already cover most starting points.

Full fine-tune

Updating every weight on your task data.

When: Rarely worth it for narrow tasks — our own cycle evidence found full fine-tuning no better than LoRA on an 18-probe suite, and reverted it.

LoRA / QLoRA

Low-rank adapters beside frozen weights; QLoRA trains them over a 4-bit base.

When: The 2026 default for in-boundary specialization — small artifact, auditable diff, trains on workstation-class hardware.

DoRA / GaLore

Weight-decomposed adapters and gradient low-rank projection — closer-to-full-rank quality at adapter-like memory.

When: When LoRA plateaus below the quality bar and you still can't afford full-rank training.

DPO / GRPO

Preference optimization and grouped-rollout RL directly on comparison data — no separate reward model.

When: After supervised tuning, to sharpen refusal behavior, formatting discipline, and reasoning-under-policy.

RLAIF / self-rewarding

AI-generated feedback replacing human labels in the alignment loop.

When: To scale alignment data past what labelers can produce — with the judge itself now a component you must evaluate.

On-policy distillation

A large teacher grades or generates targets for a small student on the student's own outputs.

When: The frontier-to-small pipeline: when a tuned small model must approach big-model behavior on one domain at a fraction of the serving cost.

Quantization (GGUF / AWQ / GPTQ / FP8)

Lower-precision weights for deployment — 4-bit typically retains ~97–99% quality (published ranges, format-dependent).

When: At deploy time, to fit the boundary's hardware; retention must be measured on your evals, not assumed from the format's reputation.

Pruning (Wanda / SparseGPT)

One-shot removal of low-salience weights.

When: When quantization alone doesn't reach the footprint target and you can re-verify quality after.

Merging (TIES / DARE / SLERP)

Combining adapters or variants in weight space without retraining.

When: To compose capabilities from separately trained adapters — cheap, but provenance of the merged artifact must stay explicit.

Continual eval-gated loop

Measure → modify → keep or discard, with every kept change justified by an eval delta.

When: Always — it is the difference between a training run and a training process. Each cycle's verdict (kept / reverted / inconclusive) is release evidence.