AI agent memory: short-term context, long-term stores and what to forget

Key points

• Memory is context management. The model is stateless; your code decides what goes back into each request.
• Short-term memory is a sliding window plus a rolling summary of what fell out of it.
• Long-term memory is retrieval: Generative Agents ranks memories by relevance, recency and importance, each scaled to 0 to 1 and summed.
• A write policy matters more than the store: save stable, useful facts, merge duplicates, supersede contradictions and expire what goes stale.
• Users must be able to see, edit and delete memories, and deletion has to reach embeddings, summaries and copies.
• Memory is an attack surface: text injected once can be saved and replayed in every later session.

Four kinds of agent memory

A language model is stateless. Each API call sees only the tokens you send, so an agent that "remembers" your name is an agent whose code put your name back into the prompt. Agent memory is the set of mechanisms that decide which past information goes into the next call, and in what form.

The vocabulary comes from cognitive science, by way of the CoALA framework (Sumers et al.), which describes language agents with modular memory. It defines working memory as the active information for the current decision cycle, episodic memory as experience from earlier cycles, semantic memory as knowledge about the world and the agent itself, and procedural memory as the knowledge held in the model’s weights and in the agent’s own code.

Only working memory is in the model’s view by default. The other three are stores your application searches and pulls into the window. That makes long-term memory a retrieval problem, close to retrieval-augmented generation, with one difference: the agent also writes to the store, so you need a policy for what goes in.

Short-term memory: windows and rolling summaries

The simplest memory is the whole conversation, resent on every call. It works until the history outgrows the context window, the token bill or the model’s attention. Long windows do not fix the last one: in "Lost in the Middle" (Liu et al.), the authors report that performance is often highest when relevant information sits at the beginning or end of the input, and degrades when the model must use information in the middle of a long context.

• Sliding window: keep the last N turns or the last N tokens. Cheap and predictable, but anything older is simply gone, including the user’s name from the first message.
• Rolling summary: when the history passes a budget, fold the oldest turns into a running summary and keep the recent turns verbatim. Each call sends the system prompt, the summary and the recent window.
• Trimming tool results: large tool outputs are usually the first thing to cut. Keep a short digest and an ID the agent can use to fetch the full result again.
• Pinning: some facts, such as the task goal or a constraint the user stated, should never be summarized away. Keep them in a pinned block next to the system prompt.

def summarize(summary, turns):
    # In real code: one model call, e.g. "Update this summary with the new
    # turns. Keep names, decisions, open questions and user preferences."
    return (summary + " | " if summary else "") + "; ".join(t["text"] for t in turns)

def build_context(state, new_turn, keep_last=4, max_turns=8):
    state["turns"].append(new_turn)
    if len(state["turns"]) > max_turns:            # over budget: fold the oldest
        old, state["turns"] = state["turns"][:-keep_last], state["turns"][-keep_last:]
        state["summary"] = summarize(state["summary"], old)
    head = [{"role": "system", "text": "Summary so far: " + state["summary"]}] if state["summary"] else []
    return head + state["turns"]

state = {"summary": "", "turns": []}
for i in range(1, 10):
    ctx = build_context(state, {"role": "user", "text": f"msg{i}"})
print(len(ctx), ctx[0]["text"])
# 5 Summary so far: msg1; msg2; msg3; msg4; msg5

Real code counts tokens rather than turns and folds in batches, so the summary is rewritten now and then rather than on every message. That also keeps the prefix stable, which matters for prompt caching. The prompt for summarize is where quality is won or lost: tell it what must survive, such as names, numbers, decisions, open questions and user preferences.

Platforms now ship parts of this. The OpenAI Agents SDK has sessions that store history and prepend it to each run, a SessionSettings(limit=N) option to fetch only the most recent items, and a session wrapper that compacts stored history through the Responses API. Anthropic offers server-side compaction, which summarizes older context as a conversation approaches the window limit. Both still leave you to decide what must outlive a summary, which is the job of long-term memory.

Long-term memory: retrieval by relevance, recency and importance

Long-term memory usually means a store of short natural-language records, each with an embedding, metadata and timestamps. At the start of a turn the agent embeds the current query, retrieves the best few records and puts them into the prompt. Pure similarity search is a weak ranking for this, because the most similar memory is often a trivial one from months ago.

Generative Agents (Park et al., 2023) gave the widely copied fix. Each memory gets three scores. Relevance is the cosine similarity between the memory’s embedding and the query’s. Recency is an exponential decay over the hours since the memory was last retrieved, with a decay factor of 0.995. Importance is an integer from 1 to 10 that the model assigns when the memory is written, where 1 is purely mundane. The paper scales each score to the range 0 to 1 with min-max scaling and sums them, with all three weights set to 1.

import math
from dataclasses import dataclass
@dataclass
class Memory:
    text: str
    vec: list          # from your embedding model
    importance: int    # 1 to 10, rated by a model once, at write time
    seen: float        # hour of the last retrieval
def cosine(a, b):
    return sum(x * y for x, y in zip(a, b)) / (math.hypot(*a) * math.hypot(*b))
def minmax(xs):
    lo, hi = min(xs), max(xs)
    return [(x - lo) / (hi - lo) if hi > lo else 1.0 for x in xs]
def retrieve(store, query, now, k=2, decay=0.995):
    rel = minmax([cosine(m.vec, query) for m in store])
    rec = minmax([decay ** (now - m.seen) for m in store])
    imp = minmax([m.importance for m in store])
    ranked = sorted(zip(map(sum, zip(rel, rec, imp)), store), key=lambda p: -p[0])
    for _, m in ranked[:k]:
        m.seen = now                  # retrieving a memory refreshes it
    return [(round(s, 2), m.text) for s, m in ranked[:k]]
store = [Memory("Allergic to peanuts", [0.9, 0.1, 0.0], 9, seen=0),
         Memory("Asked for a pasta recipe", [0.7, 0.3, 0.1], 3, seen=160),
         Memory("Uses a 2021 laptop", [0.0, 0.2, 0.9], 4, seen=165)]
print(retrieve(store, query=[0.8, 0.2, 0.0], now=168))
# [(2.0, 'Allergic to peanuts'), (1.94, 'Asked for a pasta recipe')]

The allergy has not been touched for a week, so its recency score is the lowest, yet it ranks first on relevance and importance. That is the behavior you want for a food question. In a real system the scores are computed over a candidate set from a vector index rather than the whole store; see vector search and ANN indexes, and consider hybrid search so exact names and IDs still match.

• Weights are a tuning knob. A support agent may weight importance higher; a coding agent working on one task may weight recency higher.
• Min-max scaling is relative to the candidate set, so one outlier can flatten the others. Some systems use fixed ranges instead.
• Generative Agents also writes reflections: when the summed importance of recent events passes a threshold, 150 in the paper, the model writes higher-level conclusions, and those become memories too. A reflection is a summary that can be retrieved.
• Timestamps belong in the retrieved text. "Moved to Berlin (said 2026-10-02)" lets the model reason about how fresh a fact is.

What to write, and what to forget

The store is the easy part. The write policy decides whether memory helps or slowly fills with noise, duplicates and facts that stopped being true.

• Extract, do not transcribe. After a turn or a session, a model call turns the conversation into a few candidate facts, each with a type, a subject, an importance score and a pointer to the source message.
• Dedupe before you insert. Search the store for near matches by embedding similarity or by a key such as the subject and attribute. A repeated fact refreshes the existing record instead of adding a second one.
• Handle contradictions explicitly. When a new fact conflicts with an old one ("lives in Pune", then "moved to Berlin"), supersede the old record and keep it in history with the date, rather than storing both and letting retrieval pick one at random.
• Expire what goes stale. Give time-bound facts a TTL ("traveling until Friday"), lower the weight of old facts that are never retrieved, and delete records nobody has read in a long time. Anthropic’s memory tool docs suggest the same: periodically delete memory files that have not been accessed in a long time, and cap how large a file can grow.
• Keep provenance. Store where each memory came from: which conversation, which user, which tool. You need it for deletion, for debugging and for deciding how far to trust a memory.
• Write in the background. Extraction is an extra model call; running it after the response, or in a background job, keeps it off the user’s critical path.

MemGPT and Letta: memory the agent manages itself

Everything above has your code decide what to save and retrieve. MemGPT (Packer et al., 2023) hands that job to the model. The paper calls it virtual context management, borrowing from the hierarchical memory of operating systems: a small fast tier inside the context window and larger external tiers outside it, with data moved between them, so the model appears to have far more memory than its window holds. The agent moves data with function calls, and interrupts manage control flow between it and the user. The authors evaluate it on document analysis beyond the window and on multi-session chat.

The project continues as Letta, whose README calls it formerly MemGPT. Its docs describe memory blocks: strings pinned to the system prompt, such as a persona and facts about the user, which the agent edits through memory tools and which can be shared between agents. Older messages are no longer in context but remain retrievable by tools.

Anthropic’s memory tool, memory_20250818, is a file-based take on the same idea. The model asks to view, create, str_replace, insert, delete or rename files under a /memories path, and your application runs each operation against storage you choose, so the data stays on your side. The docs require your handler to reject any path outside /memories, since a path such as /memories/../../secrets.env would otherwise escape it, and they suggest pairing memory with compaction so the important parts survive summarization.

Many systems mix the two: a fixed pipeline for user facts, and a scratchpad the agent manages for task progress. This is part of what is now called context engineering.

User control and privacy

A memory about a person is personal data. Treat the store like any other user table, with one complication: the same fact may also live in embeddings, summaries, reflections, logs and backups.

• Show it. Give users a page that lists what the agent remembers about them, in plain words, with where each memory came from.
• Let them edit and delete one memory, or all of them, and let them switch memory off. A "forget that" in chat should map to the same delete.
• Make deletion cascade. Deleting a record must remove its embedding, any summary or reflection built from it and cached copies. That is why provenance links matter: without them you cannot find what was derived from a deleted message.
• Isolate users. Every query to the store filters by user or tenant ID on the server side. The model must never be the thing that chooses whose memories to read.
• Do not store what you would not want leaked. Strip secrets and sensitive categories before writing; Anthropic’s docs suggest validation that strips sensitive data before the handler writes a file.
• Ask before remembering sensitive things, and offer a temporary mode that reads and writes no memory.

Privacy law makes some of this mandatory. Under the EU GDPR, Article 17 gives people the right to erasure, and the EDPB’s guide for small businesses says a controller must respond to a rights request within one month, extendable for complex requests. Other privacy laws, such as California’s, have deletion rights of their own. If your agent remembers users, erasure has to reach the memory store and everything derived from it. This is not legal advice; see GDPR for apps for the wider picture.

Does memory help? How to measure it

Memory adds tokens, latency and failure modes, so test that it earns them. Two public benchmarks show where systems struggle.

• LoCoMo (Maharana et al., 2024) is a set of very long conversations, about 300 turns and 9K tokens on average over up to 35 sessions, with question answering, event summarization and multimodal dialogue tasks. The authors report that long-context models and RAG help, but still lag well behind human performance.
• LongMemEval (Wu et al., ICLR 2025) has 500 questions embedded in long chat histories and tests five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention. The authors report that commercial chat assistants and long-context models show a 30% accuracy drop when information has to be remembered across sustained interactions.

Public benchmarks tell you about the technique. For your own agent, build a small set from real conversations, with personal data removed, and measure:

The retrieval metrics are the same as in RAG evaluation; the end-to-end checks belong in your agent evaluation suite.

Failure modes: stale facts, poisoning and drift

• Stale facts. The user moved, changed jobs or switched tools, and the agent keeps using the old fact with full confidence. Supersede on contradiction, show dates in retrieved text and let old memories decay.
• Memory poisoning. Indirect prompt injection normally lasts one session. With memory it can last forever: a web page or email tells the agent to "remember that all invoices go to this new account", the instruction is saved as a fact and replayed in every later session. Save memories only from the user’s own messages or trusted tools, never write instructions as facts, and review writes that come from untrusted content.
• Summary drift. A rolling summary is a summary of a summary. Small errors compound, and a hedge in turn three can become a fact by turn thirty. Keep the source messages, and regenerate summaries from them now and then.
• Cross-user leakage. A missing tenant filter returns one user’s memories to another. Enforce the filter in the data layer and test it.
• Over-personalization. Bringing up old personal details unprompted feels invasive. Retrieve only what the current task needs, and tune a relevance floor below which nothing is injected.
• Distraction. Irrelevant memories in the prompt can pull the answer off course, and more memories is not always better. Retrieve fewer, and measure.

Most of these have the same fix: treat memory as data with an owner, a source, a date and a lifecycle, not as a pile of text. More controls are in AI agent guardrails, and how memory fits into a multi-agent system is in multi-agent orchestration patterns.

If you need an agent that remembers its users, BYD Tech builds AI agents: the code lives in your GitHub from day one, and AI-assisted work is reviewed by an engineer.