AI Agent Memory and Context Management

Step 1: Distinguish short-term and long-term memory

Short-term memory is the conversation history in the current run: messages, tool calls, and results. Long-term memory lives outside the model, in files or databases, and persists across sessions, such as user preferences or project facts. The agent reads from it when needed and writes to it when it learns something worth keeping.

Step 2: Give the agent a memory tool

The simplest long-term memory is a pair of tools to save and read notes. The agent decides what is worth remembering. Validate what gets written and avoid storing sensitive data without a clear need.

import json, pathlib

NOTES = pathlib.Path("notes.json")

def save_note(key: str, value: str) -> str:
    data = json.loads(NOTES.read_text()) if NOTES.exists() else {}
    data[key] = value
    NOTES.write_text(json.dumps(data, indent=2))
    return "Saved note: " + key

def read_notes() -> str:
    return NOTES.read_text() if NOTES.exists() else "{}"

Step 3: Load relevant memory at the start

At the beginning of a session, read stored notes and place them in the system prompt or first message so the agent starts informed. For large memories, use retrieval, as in RAG, to load only what is relevant instead of everything.

SYSTEM = (
    "You are a helpful assistant.\n\nWhat you know about this user:\n"
    + read_notes()
)

Step 4: Control the growth of history

Long runs can be managed by summarizing older turns into a short note, trimming bulky tool results once they have been used, or starting a fresh context with a summary. If you trim messages, keep each tool request paired with its result, because the API expects them together.

Step 5: Use platform features where they exist

Some platforms offer server-side help: automatic compaction that summarizes earlier context as it nears a threshold, context editing that clears old tool results, a ready-made memory tool, and prompt caching that lowers the cost of repeated context. These are often in beta and vary by model, so check the current documentation before relying on them.