THE SHORT ANSWER
Data may be insufficient when the situation is new, samples are small, important outcomes are unmeasured, objectives conflict or values determine acceptable trade-offs. Use data with qualitative evidence, domain knowledge, scenarios and explicit judgment rather than asking one metric to decide.
Name why the data is insufficient
| Condition | Judgment required |
|---|---|
| New situation | Which past evidence is transferable? |
| Small sample | Which conclusions are too unstable? |
| Incomplete measurement | Which important outcome is invisible? |
| Competing objectives | How should trade-offs be weighted? |
| Qualitative change | What does the metric fail to describe? |
| Value choice | What should be optimized and for whom? |
A metric cannot choose its own objective
A support team can minimize handling time by ending conversations quickly. Whether that is desirable depends on accuracy, customer outcomes, staff capacity and the purpose of support. The number describes one dimension; people must define the decision.
A single summary metric can also hide distribution. An average improvement may coexist with worse outcomes for an important group.
Use evidence types for different questions
- Operational data: what happened and how often?
- Experiments or comparisons: what changed under different conditions?
- Interviews and observation: how and why did people experience it?
- Domain expertise: which mechanisms and constraints are plausible?
- Values and objectives: which trade-offs are acceptable?
Keep judgment accountable
Document where evidence ends and judgment begins. State the objective, missing information, competing values and review trigger. The Marketing Analytics guide to turning data into business decisions provides the measurement workflow; this module asks how strong the conclusion is and what else the decision requires.
Sources & further reading
- The Green Book 2026
HM Treasury and Government Finance Function. Current official guidance on objectives, options, evidence, risk, uncertainty, appraisal and evaluation. It is designed for public decisions, while the general reasoning principles are adapted here.
- Reproducibility and Replicability in Science
National Academies of Sciences, Engineering, and Medicine. A consensus report on evidence, uncertainty, transparency, reproducibility and replication. Scientific standards need proportionate adaptation outside research settings.
Examples and exercises are illustrative unless attributed to a source. No independent expert review is claimed.
A correction, a counterexample or an experience worth sharing?
Join the conversation ↗