Agent memory architecture: short-term, long-term, episodic

Most production agents combine three layers, each doing a different job. Short-term or working memory holds the current task context — what the agent is reasoning over right now — and usually lives in the model's context window: fast and highly relevant, but bounded in size and paid for on every iteration. Long-term memory persists facts and knowledge in an external store, often a vector database, and is retrieved only when relevant; this is where retrieval-augmented generation lives, and it scales recall far beyond what fits in the context window. Episodic memory records past interactions or whole runs the agent can look back on, which supports continuity and personalisation across sessions. The skill is not maximising any one layer but routing the right information to the right place: keep working memory tight, fetch long-term knowledge on demand, and write episodic records deliberately. Each layer also has different lifetime, cost and trust characteristics, which is why they are governed differently rather than treated as one undifferentiated memory.

Because anything an agent stores can be recalled later and treated as trusted context, which turns the memory write path into an attack surface rather than a neutral convenience. The dominant risk is memory poisoning: if writes are not validated, false or malicious content can persist in long-term or episodic memory and quietly bias many future runs, long after the original event, and because it is recalled as established context it is hard to detect. The second risk is data exposure: over-broad retention of personal data raises obligations under the GDPR and widens the blast radius of any breach, because a store full of remembered customer data is exactly what an attacker wants. There is also the inter-session angle — memory is one of the few ways a single compromise can outlive the session that caused it. So memory has to be governed, not merely optimised for recall: validate what enters it, segment by trust level so untrusted sources cannot become trusted context, set lifetimes, and apply least privilege to who and what may write.

Less than most teams assume, because the instinct to keep more usually makes the agent worse, not better. Every token of working memory is paid for on every iteration, so a bloated context is expensive, and worse, it dilutes the signal the model needs — burying the few relevant facts among many irrelevant ones measurably degrades reasoning. The better pattern is selective by design: keep working memory tight and scoped to the current step, retrieve long-term knowledge on demand rather than preloading everything, and store episodic records deliberately with an expiry rather than hoarding every interaction forever. More memory is not more capability; relevant memory is. A useful way to think about it is that the goal is recall precision, not recall volume — the agent should be able to reach the right information when it needs it, which is a routing and retrieval problem, not a storage-maximisation one. Start minimal, add a memory layer only when a concrete task failure shows it is missing, and prune aggressively.

No — RAG is one component of agent memory, specifically the long-term layer, not the whole picture. Retrieval-augmented generation is a technique for fetching relevant facts or documents from an external store, usually vector-backed, and injecting them into the context at the moment they are needed, which is exactly what long-term memory does. But an agent's memory architecture is broader: it also includes short-term working memory in the context window, which RAG does not address, and episodic memory of past interactions and runs, which is about continuity rather than knowledge lookup. Conflating the two leads to a common mistake — assuming that adding RAG gives the agent memory — when in fact you may still lack working-memory discipline or any episodic recall. The practical framing is that RAG answers what the agent knows and can look up, while working memory answers what the agent is holding right now and episodic memory answers what has happened before. Design all three deliberately; RAG is the powerful middle layer, not a substitute for the others.