Context engineering: long context and memory
Context engineering is choosing exactly what goes into the model's context window on each call: instructions, examples, retrieved documents, history and tool results. The aim is to give it what it needs and nothing that distracts it.
Why it matters
Context windows are large but not free: every token adds cost and delay, and models use information buried in the middle of very long inputs less reliably. Long chats and agents keep adding history and tool output. Deciding what to keep, summarise or drop is now one of the main skills in building reliable LLM apps.
The idea
Context engineering means deciding exactly what goes into the model's context window on each call: instructions, examples, retrieved documents, conversation history and tool results. More is not always better. Models use information in the middle of very long inputs less reliably, and every token costs time and money. For long conversations you summarise or store old turns and bring back only what matters. That is 'memory'.
Packing a small suitcase for a trip. You can't bring your whole wardrobe, so you choose what this trip needs.
A suitcase has a hard limit you can feel. A context window can hold a lot, so the danger is quieter: stuffing it full makes answers worse and slower without any error message.
How it works
- Set a token budget for the window: instructions, examples, retrieved documents, history, and room for the answer.
- Keep instructions and the key examples stable at the start (which also makes prompt caching effective).
- Retrieve only the most relevant documents, ranked, rather than everything that might help.
- For long conversations, keep the last few turns word for word and replace older ones with a running summary.
- Store long-term facts (user preferences, decisions) outside the context and bring them back only when relevant: memory.
- Trim tool outputs to what matters, such as the 5 relevant rows rather than the whole table.
- Place the most important material near the start or the end, and test with "needle in a haystack" checks at the sizes you actually use.
Key points
- More context is not always better: quality, speed and cost can all get worse.
- Information in the middle of long contexts is used less reliably ("lost in the middle").
- Summaries plus retrieval scale better than whole histories.
- Every token is billed and adds latency.
- Agents need context management most, because tool results add up fast.
Common mistakes
- Sending the entire history and every retrieved document by default.
- Assuming that fitting in the window means the model will use all of it.
- Leaving no room for the answer.
- Summarising away details the user will ask about later.
Not true: Filling the whole context window with documents still leaves the model room to answer.
In fact: The answer has to fit in the same window. If the input fills it, the reply is cut short, so keep space free for the output.
Not true: If a document fits in the context window, the model will use all of it.
In fact: Models use the start and the end of a long context best. Information in the middle is used less reliably ("lost in the middle").
Part of Unit 5: AI engineering in the free ML Foundations course.