Context engineering: long context and memory

Context engineering is choosing exactly what goes into the model's context window on each call: instructions, examples, retrieved documents, history and tool results. The aim is to give it what it needs and nothing that distracts it.

Why it matters

Context windows are large but not free: every token adds cost and delay, and models use information buried in the middle of very long inputs less reliably. Long chats and agents keep adding history and tool output. Deciding what to keep, summarise or drop is now one of the main skills in building reliable LLM apps.

The idea

Context engineering means deciding exactly what goes into the model's context window on each call: instructions, examples, retrieved documents, conversation history and tool results. More is not always better. Models use information in the middle of very long inputs less reliably, and every token costs time and money. For long conversations you summarise or store old turns and bring back only what matters. That is 'memory'.

Packing a small suitcase for a trip. You can't bring your whole wardrobe, so you choose what this trip needs.

A suitcase has a hard limit you can feel. A context window can hold a lot, so the danger is quieter: stuffing it full makes answers worse and slower without any error message.

How it works

  1. Set a token budget for the window: instructions, examples, retrieved documents, history, and room for the answer.
  2. Keep instructions and the key examples stable at the start (which also makes prompt caching effective).
  3. Retrieve only the most relevant documents, ranked, rather than everything that might help.
  4. For long conversations, keep the last few turns word for word and replace older ones with a running summary.
  5. Store long-term facts (user preferences, decisions) outside the context and bring them back only when relevant: memory.
  6. Trim tool outputs to what matters, such as the 5 relevant rows rather than the whole table.
  7. Place the most important material near the start or the end, and test with "needle in a haystack" checks at the sizes you actually use.

Key points

Common mistakes

Not true: Filling the whole context window with documents still leaves the model room to answer.
In fact: The answer has to fit in the same window. If the input fills it, the reply is cut short, so keep space free for the output.

Not true: If a document fits in the context window, the model will use all of it.
In fact: Models use the start and the end of a long context best. Information in the middle is used less reliably ("lost in the middle").

Part of Unit 5: AI engineering in the free ML Foundations course.