Agent Memory skill for Codex

Done well means: every memory has a defined type, source and lifetime; the context window holds only what the current turn needs; long-term memories are written deliberately (not every message), retrieved by relevance and recency, attributed to their source, scoped to one user or tenant, and deletable by the user.

Memory types

Type	Holds	Lifetime	Store
Working (context window)	Current turn, recent messages, tool results	One call	Prompt assembly
Session / thread	Full conversation, agent state	Conversation	Checkpointer or messages table
Session summary	Rolling summary of older turns	Conversation	Same row as session
Profile (semantic)	Stable facts: name, timezone, preferences, plan	Until changed	Key-value / JSON per user
Episodic	Past events and outcomes (“ticket 812 solved by clearing cache”)	Months, with decay	Vector-indexed table
Procedural	Learned instructions (“user wants diffs, not full files”)	Until revoked	Profile or system-prompt fragment, reviewed

Start with session state plus a small profile. Add episodic memory only when evals show the agent needs past episodes.

Workflow

- [ ] 1. List what the agent must remember, why, and for how long (with the product owner)
- [ ] 2. Assign each item a type from the table, a scope (user, tenant, global) and a TTL
- [ ] 3. Implement short-term handling: token budget, trimming, rolling summary
- [ ] 4. Implement write policy for long-term memory (extraction + dedupe)
- [ ] 5. Implement retrieval: filters, similarity, recency, cap on injected tokens
- [ ] 6. Add user controls: view, edit, delete, opt out; retention job
- [ ] 7. Evaluate: recall of needed facts, wrong-memory rate, token cost per turn

Check AGENTS.md and existing storage first; memory usually belongs in the app’s primary database, not a new service.

Short-term: fit the window

Budget the prompt explicitly, for example: system 1.5k, profile 0.5k, retrieved memories 1k, retrieved documents 4k, recent messages 6k, reserve for output 2k.

def build_history(messages: list[dict], summary: str | None, budget_tokens: int) -> list[dict]:
    kept, used = [], 0
    for m in reversed(messages):                    # newest first
        t = count_tokens(m)
        if used + t > budget_tokens:
            break
        kept.append(m)
        used += t
    kept.reverse()
    kept = drop_orphan_tool_results(kept)           # never start with a tool result whose call was cut
    prefix = [{"role": "system", "content": f"Summary of earlier conversation:\n{summary}"}] if summary else []
    return prefix + kept

Rolling summary: when unsummarised history exceeds a threshold (for example 70% of its budget), summarise the oldest chunk together with the previous summary into a new summary. Keep decisions, open tasks, names, numbers and commitments; drop pleasantries. Store the summary with the index of the last message it covers so it is never applied twice.

Framework equivalents: LangGraph checkpointers plus summarisation middleware or a summary node ($langgraph-agents); OpenAI Responses previous_response_id or conversation state ($openai-responses-api). Server-side state still needs a token strategy for long threads.

Long-term: write policy

Writing every message to a vector store produces noise that later poisons answers. Write when:

• the user states a durable fact or preference (“I’m on the EU data plan”, “use metric units”),
• the user explicitly asks the agent to remember something,
• a task concludes with a reusable outcome (resolution, decision, artifact location).

EXTRACT_PROMPT = """From the conversation below, extract durable facts about the user or their work
that will matter in future sessions. Ignore one-off requests and anything the user asked not to keep.
Return JSON: {"memories": [{"kind": "preference|fact|episode", "text": "...", "confidence": 0-1}]}"""

def write_memories(user_id: str, convo: list[dict]) -> None:
    for m in extract(EXTRACT_PROMPT, convo).memories:
        if m.confidence < 0.7 or contains_secret(m.text):
            continue
        existing = nearest_memory(user_id, m.text, threshold=0.9)
        if existing:
            update_memory(existing.id, text=m.text, source_turn=convo[-1]["id"])   # supersede, don't duplicate
        else:
            insert_memory(user_id, m, source_turn=convo[-1]["id"])

Run extraction asynchronously after the turn (background job), not in the response path.

Storage schema (Postgres + pgvector)

CREATE TABLE agent_memories (
  id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id    uuid NOT NULL,
  user_id      uuid NOT NULL,
  kind         text NOT NULL CHECK (kind IN ('preference','fact','episode','procedure')),
  text         text NOT NULL,
  embedding    vector(1536) NOT NULL,
  source_turn  text,                       -- message or run ID for attribution
  confidence   real NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  last_used_at timestamptz,
  expires_at   timestamptz,
  superseded_by uuid REFERENCES agent_memories(id)
);
CREATE INDEX ON agent_memories (tenant_id, user_id) WHERE superseded_by IS NULL;

Index and filtered-query tuning: $pgvector-search.

Retrieval

SELECT id, kind, text,
       (1 - (embedding <=> $3)) * 0.8
       + 0.2 * exp(-extract(epoch FROM now() - coalesce(last_used_at, created_at)) / (86400 * 30)) AS score
FROM agent_memories
WHERE tenant_id = $1 AND user_id = $2
  AND superseded_by IS NULL
  AND (expires_at IS NULL OR expires_at > now())
ORDER BY score DESC
LIMIT 8;

• Always load the profile (small, structured); retrieve episodic memories by similarity to the current request plus a recency term. Weights are a starting point; tune on evals.
• Inject memories in a labelled block with dates: - (2026-03-02, preference) Prefers metric units. Tell the model memories may be outdated and the current message wins on conflict.
• Update last_used_at for memories that were injected and cited.

Privacy and safety

• Scope every query by tenant and user server-side; a memory leak across users is a data breach.
• Do not store secrets, credentials, payment data, health or other special-category data unless the product explicitly requires it and the user consented. Filter before writing.
• Provide a “what do you remember about me” view and per-item delete; deletion must remove the row and embedding, not just hide it. Confirm with the user before bulk deletion.
• Memories written from untrusted content (web pages, emails) can carry injected instructions that persist across sessions. Only write memories from user statements or verified outcomes, and treat retrieved memories as data ($prompt-injection-defense).
• Retention: a scheduled job expires episodes after the agreed TTL; document it.

Gotchas

• Contradictions: “lives in Berlin” and later “moved to Lisbon” both retrieved. Supersede on write using similarity + kind, and prefer newest on read.
• Summaries drift: repeated re-summarisation loses numbers and names. Keep key entities in a structured field next to the prose summary.
• Over-personalisation: injecting every memory every turn wastes tokens and causes odd replies. Cap count and require a relevance floor.
• Truncating history in the middle of a tool call/result pair breaks most chat APIs.
• Evaluate with multi-session scripts: session 1 states a preference, session 3 should use it.

Verification

• Multi-session eval (20+ scripts): needed-fact recall, wrong or stale memory usage rate, tokens added per turn.
• Isolation test: user B never retrieves user A’s memories, including with near-identical text.
• Deletion test: after delete, the memory is absent from storage and from the next retrieval.
• Long-thread test: a 200-turn conversation stays within the token budget without API errors.

Design summary template

**Short-term:** last 6k tokens of messages + rolling summary (trigger at 70%), tool pairs kept intact
**Profile:** user_profiles JSONB (units, timezone, plan); always injected (< 300 tokens)
**Episodic:** agent_memories (pgvector), written by async extractor, conf >= 0.7, dedupe at 0.9 similarity
**Retrieval:** top 8 by 0.8 similarity + 0.2 recency (30-day decay), relevance floor 0.3
**Controls:** memory viewer, per-item delete, 180-day episode TTL, no secrets or payment data stored
**Eval:** 24 multi-session scripts, fact recall 0.88, stale-memory errors 2/24