Write a task specification before comparing models
- Outcome: what useful result should exist?
- Acceptance: what must be correct, complete and well formed?
- Risk: what happens when the result is wrong?
- Environment: what languages, formats, tools and context appear in practice?
- Operations: what latency, volume and budget constraints matter?
A general benchmark cannot represent all of those conditions. Build a representative evaluation set from real patterns, remove sensitive data where necessary, and include failures that matter to the organisation.
Use a threshold, then compare efficiency
| Dimension | Example measure | Decision use |
|---|---|---|
| Quality | Accepted without correction | Reject options below the required threshold |
| Safety | Severe failure rate | Set stricter gates for consequential tasks |
| Latency | Time to accepted result | Check real workflow, not isolated response time |
| Cost | Total cost per accepted task | Include retries, tools and review |
| Operations | Rate-limit and format reliability | Test at expected volume |
Evidence & context: Microsoft Learn
Start capable, then find the boundary
During design, a capable model can help establish what good performance looks like. Once the task and evaluation are stable, test smaller or faster options. Move down until the acceptance threshold is missed, then choose the lowest tier that passes with a margin appropriate to change and risk.
Do not make the choice permanent. Inputs drift, providers update models and prices change. Version the evaluation set and rerun it when workload, model or policy changes.
Evidence & context: OpenAI Developers · Microsoft Learn
Route only when the variation justifies it
A mixed workload may benefit from rules or a model router: routine cases go to an efficient model and uncertain cases escalate. Routing adds its own errors and operational complexity, so evaluate category-level results and fallbacks. A single model is often better when volume is small or the task is uniform.
Evidence & context: Microsoft Learn
Sources & further reading
- Models
OpenAI Developers. Official model-selection documentation, checked 13 September 2026. Product names, capabilities and prices can change; the collection uses the durable principle of matching capability to a task rather than prescribing a current model.
- Evaluate a model router
Microsoft Learn. Official guidance for evaluating routing across representative workloads using quality, cost, latency and policy criteria. It is not evidence that routing always improves results.
- Generative Artificial Intelligence Profile (NIST AI 600-1)
NIST. Risk-management guidance, including confabulation. It does not establish a universal error rate.
Examples and exercises are illustrative unless attributed to a source. No independent expert review is claimed.
A correction, a counterexample or an experience worth sharing?
Join the conversation ↗