THE SHORT ANSWER

The context window is the bounded information available to a model for an interaction. Instructions, conversation history, retrieved documents, tool results and the generated response can all draw on that capacity. More context can increase usage and latency, while irrelevant context can distract from the evidence that matters.

Treat context as a working set

Context may contain a stable system instruction, the current task, earlier turns, examples, retrieved passages and tool outputs. The window is a capacity boundary around that working set. It is not long-term memory, a truth filter or a reason to attach every available document.

Retrieval can select material for a request, but retrieval quality and source quality still need evaluation. A relevant-looking passage can be outdated or insufficient.

Evidence & context: Lewis and colleagues, 2020 · NIST

Repeated context can become the dominant input

A short question sent with a long history is not a short request. Stable instructions and reference material may be processed repeatedly across a workflow. Some providers offer caching for repeated prefixes, but caching changes the price or latency of processing; it does not make irrelevant material useful.

Evidence & context: OpenAI Developers

Build context in layers

  • Task: the current objective and acceptance criteria.
  • Rules: stable constraints that apply to this task.
  • Evidence: the smallest set of authoritative passages needed for the decision.
  • State: prior decisions that remain relevant, not an unfiltered transcript.
  • Tools: concise results with provenance and error status.

Label sources and distinguish instructions from quoted content. This helps people audit the result and reduces the chance that text inside a document is mistaken for a command.

Test context changes against quality

Create a small evaluation set with easy, ambiguous and edge cases. Compare a full-context version with a selected-context version. Measure accepted answers, missing evidence, latency and total input. If quality falls, diagnose what evidence was lost rather than restoring everything by default.

Sources & further reading

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

    Lewis and colleagues, 2020. Research combining generation with retrieved material. Retrieval should not be interpreted as proof of factual accuracy.

  2. Prompt caching

    OpenAI Developers. Provider-specific guidance on reusing repeated prompt prefixes to reduce latency and input cost. Availability and implementation details vary by model and provider.

  3. Generative Artificial Intelligence Profile (NIST AI 600-1)

    NIST. Risk-management guidance, including confabulation. It does not establish a universal error rate.

Examples and exercises are illustrative unless attributed to a source. No independent expert review is claimed.

A correction, a counterexample or an experience worth sharing?

Join the conversation ↗