Optimise the right layer
| Practice | Primary question | Failure mode |
|---|---|---|
| Token optimisation | Can unnecessary processing be removed? | Removing evidence or causing retries |
| Prompt optimisation | Can the model understand and complete the task more reliably? | Polishing wording around a poorly chosen task |
| Workflow optimisation | Should this step exist, and what should happen around it? | Adding complexity whose overhead exceeds its benefit |
Calculate the successful-task cost
Use: total task cost = model usage + tools + infrastructure + human review + failure cost. Then use: cost per accepted task = total task cost ÷ accepted tasks. These are management formulas, not provider billing formulas.
Suppose a compressed prompt is cheaper per attempt but passes 70 of 100 cases, while a clearer version costs slightly more and passes 92. The relevant comparison includes the 30 versus 8 rejected cases, their retries and review—not just the first-request token count. The numbers are illustrative.
Improve in a deliberate order
- Remove unnecessary model calls.
- Clarify the task and acceptance format.
- Choose a model using representative evaluation.
- Select relevant context and tools.
- Constrain output to what is used.
- Reduce repeated context and retry loops.
- Re-measure quality, latency and total cost.
Treat quality as a constraint
For low-risk drafting, the threshold may allow human editing. For a consequential decision, the model may only prepare evidence for an authorised person. The task design determines what 'good enough' can responsibly mean.
Evidence & context: NIST
Sources & further reading
- Prompt caching
OpenAI Developers. Provider-specific guidance on reusing repeated prompt prefixes to reduce latency and input cost. Availability and implementation details vary by model and provider.
- GenerationConfig REST resource
Google Cloud. Official documentation showing that output-token limits are a controllable generation setting. Parameter names and behaviour are provider-specific.
- Generative Artificial Intelligence Profile (NIST AI 600-1)
NIST. Risk-management guidance, including confabulation. It does not establish a universal error rate.
Examples and exercises are illustrative unless attributed to a source. No independent expert review is claimed.
A correction, a counterexample or an experience worth sharing?
Join the conversation ↗