The window is a budget

Recall: the context window is a hard token ceiling (prompt + output), it's stateless between calls, and cost/latency grow with how full it is. So every token you include is a spend against a budget — and more isn't automatically better:

• "Lost in the middle." Models attend most reliably to the start and end of the context; material buried in the middle of a very long context can be effectively ignored. Put the most important instructions and the actual question where they'll be seen.
• Dilution. Padding relevant content with marginal context lowers the signal-to-noise ratio and can degrade answers, not just cost more.

The goal isn't to fill the window — it's to put the right tokens in the right places.

Ordering and prompt caching

Render order for most APIs is roughly tools → system → messages. Two consequences:

1. Stable content first, volatile content last. Keep the frozen system prompt and deterministic tool list at the front; put per-request specifics (the user's question, timestamps, IDs) at the end.
2. Prompt caching rewards this. Caching is a prefix match: the provider caches the rendered prefix up to a marked point, so identical prefixes on later requests are served from cache at a large discount (and lower latency). Any byte change in the prefix invalidates everything after it.

from anthropic import Anthropic
client = Anthropic()

# Cache a large, stable system prompt so repeated calls reuse it cheaply.
resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=1024,
    system=[{                                   # system as a list so we can mark a cache breakpoint
        "type": "text",
        "text": LARGE_STABLE_INSTRUCTIONS,      # the big, unchanging preamble
        "cache_control": {"type": "ephemeral"}, # cache everything up to here
    }],
    messages=[{"role": "user", "content": user_question}],   # volatile part — AFTER the cached prefix
)
resp.usage.cache_read_input_tokens              # >0 on a hit; 0 means a silent invalidator broke the prefix

Long conversations: don't just append forever

A chat that appends every turn eventually approaches the window limit (and gets expensive, since you resend everything). Three tools:

• Truncation / sliding window — keep the last N turns, drop the oldest. Simple, but loses early context.
• Summarization / compaction — replace older turns with a concise summary that preserves the important facts, freeing tokens while keeping continuity. Many APIs offer server-side compaction that does this automatically as you near the limit (you must pass the compaction result back).
• Retrieval — instead of keeping everything in context, store it externally (a vector store — RAG) and pull back only what's relevant to the current turn. This is context engineering meeting RAG.

The rule of thumb: keep what's relevant now, summarize what's still useful, retrieve what you might need, drop the rest.

Memory across sessions

The context window is per-conversation and gone when the session ends. Memory persists facts across sessions — Pierre's preferences, prior decisions, project state — usually by storing them externally (a file, a database, a memory tool) and loading the relevant subset back into context at the start of a new session. Memory is not "the model remembering"; it's you re-supplying curated context. Store one durable fact per record, keep it small, and load selectively (the same relevance-over-volume principle).

Sub-agent context isolation

When a task fans out to sub-agents (a topic in the Agentic track), each sub-agent gets its own context window scoped to its subtask — it doesn't carry the orchestrator's entire history. This is deliberate context engineering: it keeps each sub-agent's window focused (better attention, lower cost) and lets them work in parallel. The orchestrator passes each sub-agent only the context it needs, and merges results back. "Give each worker exactly the context for its job, no more" is the same budget discipline applied across agents.

Recap

• The context window is a budget; models attend best to the start and end ("lost in the middle"), and padding dilutes — put the right tokens in the right places.
• Order stable-first, volatile-last and use prompt caching (prefix match) to cut repeated cost/latency; watch for silent invalidators.
• For long conversations, summarize/compact and retrieve rather than appending forever.
• Memory re-supplies curated context across sessions; sub-agents get isolated, scoped context. The through-line: relevance over volume.

from anthropic import Anthropic
client = Anthropic()

question = "What is the pass threshold for the RAG Fundamentals exam?"
relevant = "The RAG Fundamentals exam has a pass threshold of 80% and a 20-minute limit."
noise = "..." * 500  # paragraphs of unrelated product copy

def ask(context):
    resp = client.messages.create(model="claude-opus-4-8", max_tokens=100,
        messages=[{"role": "user", "content": f"Context:\n{context}\n\nQ: {question}"}])
    return resp.content[0].text, resp.usage.input_tokens

print(ask(noise + relevant + noise))   # buried in the middle — may be less reliable, costs more
print(ask(relevant))                    # curated — accurate and cheap

import datetime
from anthropic import Anthropic
client = Anthropic()

BIG = "You are an expert assistant. " + ("Follow these rules carefully. " * 400)  # >1k tokens

def call(system_text):
    resp = client.messages.create(model="claude-opus-4-8", max_tokens=50,
        system=[{"type": "text", "text": system_text, "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": "Say hi."}])
    return resp.usage.cache_read_input_tokens

print("call 1:", call(BIG))                       # 0 (writes the cache)
print("call 2:", call(BIG))                       # >0 (reads the cache) — cheaper + faster

# Break it: a changing byte in the prefix invalidates the cache.
print("call 3:", call(BIG + f" Now: {datetime.datetime.now()}"))   # back to 0 — miss

from anthropic import Anthropic
client = Anthropic()

def count(messages):
    return client.messages.count_tokens(model="claude-opus-4-8", messages=messages).input_tokens

def compact(messages, budget=2000, keep_recent=2):
    if count(messages) <= budget:
        return messages
    old, recent = messages[:-keep_recent], messages[-keep_recent:]
    convo = "\n".join(f"{m['role']}: {m['content']}" for m in old)
    summary = client.messages.create(model="claude-opus-4-8", max_tokens=300,
        messages=[{"role": "user", "content": f"Summarize this conversation, preserving key facts and decisions:\n{convo}"}]
    ).content[0].text
    # Replace the old turns with one compact summary, keep the most recent turns verbatim.
    return [{"role": "user", "content": f"[Summary of earlier conversation]\n{summary}"}] + recent

# Usage: history = compact(history) before each new call once it grows large.