Why memory is a design problem

An LLM is stateless: it only knows what's in the current context window, and that window is finite (and billed per token). Every turn you re-send the whole conversation. So on any non-trivial run you hit three walls:

• The window fills up. Long conversations or long tool outputs overflow the context limit.
• Cost and latency grow with history. Re-sending 50 turns every call is expensive and slow.
• Signal gets diluted. A wall of stale history buries the few facts that actually matter ("lost in the middle").

So "memory" isn't a feature you turn on — it's how you decide what the model sees each turn. Two horizons: short-term (this conversation) and long-term (facts that outlive it).

Short-term memory: windowing and trimming

The simplest strategy: keep only the most recent messages that fit a budget, drop the oldest. Always keep the system message and enough recent turns for coherence.

def trim_history(messages: list, max_tokens: int = 3000) -> list:
    system = messages[0]                                 # always keep the system prompt (index 0)
    rest = messages[1:]                                  # the conversational turns
    # Drop oldest turns until the remaining ones fit the token budget:
    while sum(count_tokens(m["content"]) for m in rest) > max_tokens:
        rest.pop(0)                                      # remove the OLDEST turn (like arr.shift())
    return [system] + rest                               # system + the recent window that fits

Cheap and predictable — but it forgets anything that scrolled out of the window. That's fine for a customer-support chat, fatal for "earlier you told me my account id".

Short-term memory: summarization

Instead of dropping old turns, compress them. When history gets long, summarize the older portion into a compact running summary and keep only that plus the recent turns:

def summarize_memory(messages: list, keep_recent: int = 6) -> list:
    system = messages[0]
    old, recent = messages[1:-keep_recent], messages[-keep_recent:]   # split: old to compress, recent to keep verbatim
    if not old:                                          # nothing old enough to summarize yet
        return messages
    # Ask the model to distill the old turns into durable facts/decisions:
    summary = llm(f"Summarize the key facts and decisions from this conversation:\n{format(old)}")
    summary_msg = {"role": "system", "content": f"Summary of earlier conversation: {summary}"}
    return [system, summary_msg] + recent               # system + running summary + verbatim recent turns

This preserves the gist of a long conversation at a fixed token cost, at the price of an extra LLM call and some lossiness. Most production chat agents use a hybrid: summary of old + verbatim recent window.

Long-term memory: retrieval

Some facts must outlive the conversation entirely — user preferences, past decisions, profile data. You don't keep those in the message list at all; you store them externally and retrieve the relevant ones each turn. This is exactly RAG (Track 3), pointed at memories instead of documents:

# WRITE: when something worth remembering happens, embed and store it.
def remember(fact: str, user_id: str):
    memory_store.add(embed(fact), text=fact, user_id=user_id)   # a vector store keyed by user

# READ: at each turn, retrieve the few memories relevant to the current message and inject them.
def recall(query: str, user_id: str, k: int = 3) -> list[str]:
    return memory_store.search(embed(query), user_id=user_id, k=k)   # top-k relevant memories for THIS user

def build_context(user_message: str, user_id: str) -> list:
    memories = recall(user_message, user_id)                    # e.g. ["prefers metric units", "based in Paris"]
    system = {"role": "system",
              "content": "Relevant memory:\n" + "\n".join(memories)}   # inject only what's relevant NOW
    return [system, {"role": "user", "content": user_message}]

Key insight: you retrieve only the memories relevant to the current turn, so long-term memory scales to thousands of facts without bloating context — the model sees the 3 that matter, not all 3,000. (Libraries like LangGraph's store, Mem0, and Letta package this.)

Multi-step design: sub-agents for context isolation

On a big multi-step task, stuffing every step's tool output into one agent's context poisons it — irrelevant detail piles up, cost climbs, focus drifts. The fix is context isolation via sub-agents: delegate a subtask to a separate agent with its own fresh context, and return only the distilled result to the main agent.

def research_subagent(subtopic: str) -> str:
    # A SEPARATE agent with its OWN clean message list — its intermediate steps never touch the main context.
    messages = [{"role": "system", "content": "Research the subtopic; return a 3-line summary only."},
                {"role": "user", "content": subtopic}]
    return run_agent(messages)                            # does its own tool calls internally…
    # …and returns ONLY the summary — the caller never sees its 20 intermediate tool observations.

def orchestrator(task: str) -> str:
    subtopics = plan(task)                                # break the task into independent pieces
    # Each sub-agent works in isolation; the main agent only collects their compact results:
    findings = [research_subagent(st) for st in subtopics]   # (run concurrently in practice)
    return synthesize(task, findings)                    # combine the distilled outputs into the final answer

The main agent's context stays small and on-task; each sub-agent's messy intermediate work is thrown away after it returns its summary. This is how deep-research and multi-step agents stay coherent — and it's the supervisor/worker shape the next topic (Multi-agent patterns) formalizes. It also connects back to LangGraph: sub-agents are often nodes with their own scoped state.

Choosing a strategy

• Windowing — simple chats where old context is safely disposable.
• Summarization — long single conversations where the gist must persist at bounded cost.
• Retrieval (long-term) — facts that outlive the conversation or exceed any window (preferences, history).
• Sub-agent isolation — multi-step tasks where per-step detail would poison one shared context.

They compose: a production assistant often runs a summarized short-term window + retrieved long-term memories + sub-agents for heavy subtasks.

Pitfalls

• Unbounded history. Re-sending the full transcript every turn overflows the window and inflates cost — always trim or summarize.
• Dropping facts you needed. Pure windowing forgets; use summarization or retrieval for anything that must persist.
• Storing everything long-term. Remember selectively — a vector store of every message is noisy; store durable facts/decisions.
• One giant context for a multi-step task. Isolate subtasks in sub-agents and return summaries, or the main context degrades.
• Never forgetting (privacy). Long-term memory holds user data — support deletion and scope it per user (Security/Governance track).

Recap

• The context window is finite and billed, so memory is about deciding what the model sees each turn.
• Short-term: window/trim (simple, forgets) or summarize (compresses the gist at bounded cost); hybrids are common.
• Long-term: store facts externally and retrieve the relevant few per turn — memory as RAG, scaling past any window.
• Multi-step: use sub-agents to isolate context, returning only distilled results so the main agent stays focused and cheap.
• Strategies compose; pick per need and mind privacy/deletion for stored memory.
• Worked examples map to the exercises: trimming (Ex 1), summarization (Ex 2), retrieval memory (Ex 3), sub-agent isolation (Ex 4).

def count_tokens(t): return len(t.split())   # rough proxy

def trim_history(messages, max_tokens=200):
    system, rest = messages[0], messages[1:]
    while sum(count_tokens(m["content"]) for m in rest) > max_tokens:
        rest.pop(0)          # drop the oldest turn
    return [system] + rest

# After many turns, the early "account id is 7788" message has been popped:
trimmed = trim_history(long_conversation)
assert not any("7788" in m["content"] for m in trimmed)   # the fact scrolled out and is forgotten

def summarize_memory(messages, keep_recent=6):
    system, old, recent = messages[0], messages[1:-keep_recent], messages[-keep_recent:]
    if not old:
        return messages
    summary = llm("Summarize the key facts and decisions:\n" +
                  "\n".join(m["content"] for m in old))
    return [system, {"role":"system","content":f"Summary: {summary}"}] + recent

compact = summarize_memory(long_conversation)
# The summary should mention the account id even though the original turn was compressed away:
assert any("7788" in m["content"] for m in compact)

facts = ["prefers metric units", "based in Paris", "account id 7788",
         "signed up in 2024", "dislikes long emails"]
for f in facts:
    memory_store.add(embed(f), text=f)

def recall(query, k=2):
    return memory_store.search(embed(query), k=k)

print(recall("what's the weather like?"))   # -> ['based in Paris', 'prefers metric units'] — NOT the account id

def subagent(subtask):
    msgs = [{"role":"system","content":"Research this; return a 3-line summary ONLY."},
            {"role":"user","content":subtask}]
    return run_agent(msgs)          # its own fresh context; does tool calls internally

def orchestrator(task):
    subtasks = ["competitor A pricing & security", "competitor B pricing & security"]
    findings = [subagent(st) for st in subtasks]     # each returns only a short summary
    return synthesize(task, findings)

# The orchestrator only ever holds `findings` (short summaries) — never the sub-agents' 20 tool observations.