TL;DR: Agent memory is not a retrieval feature; it is infrastructure. Keep evidence durable, then compute versioned understanding from it. Compile a bounded, policy-filtered context for each turn instead of loading “relevant memories.” One substrate can serve both a child’s companion and a personal assistant when policy lives in admission, promotion, and compilation—not merely in storage.
Most AI products treat memory as a retrieval problem: save some text, create an embedding, and fetch the nearest matches during the next conversation. That works for demos. It is not enough for an agent expected to develop a durable understanding of a person.
Here is what that design can do in production. A child tells an AI companion, mid-game, “I hate this game.” A naive system embeds the sentence, and “hates games” keeps surfacing as durable context—one frustrated moment promoted to a personality trait. Meanwhile, a household assistant retrieves “buys oat milk weekly” long after the family switched back to dairy, because similarity search does not know when to stop believing something.
Both failures have the same shape: the system stored text when it needed to maintain an understanding.
I arrived at this problem through two very different products.
Antara is a personal assistant that can eventually work across conversations, email, calendars, documents, browsing, commerce, and personal finance. HeyAppu.ai is a deployed AI companion designed for children aged three to six. Its companion character, Appu, needs continuity across play, learning, emotions, routines, relationships, and parent-provided context.
One assistant may remember a household's grocery habits; the other may remember that a child enjoys panda stories and became frustrated when a game ended. Their safety policies are different, but their underlying memory problem is remarkably similar.
The system must turn a large stream of experience into a small, relevant, revisable understanding—and help the agent learn how to act, not merely what is true.
What HeyAppu.ai already taught us
HeyAppu.ai already implements an early form of this architecture.
When a conversation ends, an analyzer conservatively extracts high-value episodic memories and candidate profile facts. A typical record is structured rather than saved as an unqualified sentence:
memory:
claim: the child asked Appu to look like a panda today
kind: episode
source: child conversation
evidence: message and session reference
confidence: high
importance: medium
scope: child
horizon: weeksProfile facts remain distinct from episodes because “the child likes pandas” is different from “the child asked Appu to look like a panda today.”
Retrieval is routed between profile lookup, recent episodic recall, and deeper historical recall. A working-memory payload removes duplicates, prioritizes pinned and continuity-related events, and limits what enters the prompt. Semantic retrieval uses embeddings stored alongside memory in CockroachDB.
Maintenance happens outside the live conversation. Repeated episodes are consolidated; canonical memories gain reinforcement metadata; stronger patterns can become reflective memories or profile candidates; stale low-value episodes decay; merged or old memories eventually leave active retrieval without erasing history.
This is already beyond ordinary vector search. We do not yet have rigorous production evaluations showing how often this changes an interaction or how much history it compresses; building that evaluation suite is part of the work, not a result to imply prematurely. The implementation also reveals gaps in source weighting, provenance, conflict resolution, retrieval-aware decay, operational visibility, and procedural learning.
The shared memory model
The central design principle is:
Evidence is durable. Understanding is computed, versioned, and rebuildable.
Here, “rebuildable” does not mean that replaying nondeterministic model calls will reproduce identical bytes. It means that a versioned pipeline can re-derive a functionally equivalent—and inspectably different—understanding from authoritative evidence. Model versions, prompts, policies, and derivation lineage must therefore be recorded. User corrections must enter the journal as first-class evidence; otherwise a rebuild could silently resurrect the belief the user already corrected.
A useful memory platform needs four kinds of record, from rawest to most distilled: evidence, beliefs, episodes, and procedures.
1. Experience and evidence
These are meaningful events admitted from conversations, connected services, tools, and the environment. They are not raw logs.
A completed grocery order can be evidence. Every product rendered on the page is telemetry. A child correcting Appu can be evidence. Every audio frame is not.
The child’s panda request is admitted because it reflects user agency, is novel, and may matter later. The surrounding forty minutes of audio are not. Those are three admission signals among others; provider identifiers and content hashes then make ingestion idempotent.
2. Semantic understanding
These are current facts, preferences, routines, relationships, constraints, and inferred beliefs.
After the third panda request in two weeks, the system can propose a belief:
belief:
claim: the child likes pandas
basis: 3 episodes across 2 weeks
confidence: moderate
status: inferred
valid_from: first supporting episode
contradicting_evidence: noneIf a parent later confirms the preference, source authority upgrades the belief. A claim also retains its validity window, sensitivity, lifecycle state, and links to supporting or contradicting evidence. New evidence should strengthen, weaken, supersede, or expire beliefs rather than create an ever-growing pile of contradictions.
Categories should not be fixed in advance. A future task determines whether a purchasing cadence, a learning interest, a family relationship, or an emotional pattern is relevant.
3. Episodic memory
Episodic memory preserves what happened in a particular situation. It is valuable for continuity, explanation, reflection, and learning from outcomes.
The individual panda episodes remain retrievable for continuity—“remember when I was a panda yesterday?”—but can eventually be consolidated behind the belief they produced. For Antara, an episode might instead capture a failed Milkbasket authentication attempt and its eventual resolution. Episodes are not necessarily permanent; they can be consolidated, archived, or retained as evidence behind stronger knowledge.
4. Procedural memory
Actions belong in an audit journal first. Successful and unsuccessful action trajectories can later produce reusable procedures.
For example, Antara discovered that Milkbasket's GraphQL API accepted requests from https://milkbasket.com but rejected the www origin. The raw trajectory contains navigation, failure, diagnosis, origin change, and outcome. The reusable procedure is much smaller:
procedure:
trigger: authenticating to Milkbasket
strategy: use the apex origin, not www
evidence: successful diagnosis and execution
safety: revalidate before consequential action
revalidate: periodically, and on first failureA procedure carries enough information to know when to use it and when to stop trusting it. Not every action deserves promotion: repeated success is strong evidence, although a deterministic diagnosis may justify faster promotion. The Appu-side equivalent might be an interaction strategy such as “opening with a callback to yesterday’s game reliably re-engages this child.”
This distinction matters: an audit record says what the agent did; an episode says what happened; procedural memory says what the agent should try next time.
Context is compiled, not loaded
No agent should load a user's complete memory. A context compiler should construct a bounded package for each task.
It starts with the request, active task, identity, available tools, and token budget. It determines what the task is about, retrieves candidate beliefs, episodes, evidence, and procedures, then applies policy before ranking them.
Policy removes anything outside the subject boundary, revoked by its source, or inappropriate for this task. Ranking then emphasizes relevance, reliability, and timeliness. The final package clearly separates what is known, inferred, recent, and procedurally useful.
For Antara preparing groceries, this might include household consumption, recurring products, accepted substitutions, recent orders, and the correct provider procedure. For Appu beginning a story, the compiled package might say: likes pandas (belief, parent-confirmed); was a panda in yesterday’s story (recent episode); callbacks re-engage (procedure). Perhaps forty tokens now stand in for a month of sessions.
One substrate, two very different agents
The same memory infrastructure should not imply identical agent behavior.
Antara is expected to integrate broad personal context. Connecting Gmail, Calendar, Docs, or a grocery account allows accessible information to contribute to the user's model, subject to source permissions, security boundaries, retention, and deletion.
Appu operates in a higher-sensitivity child context. Parent input, tool evidence, child statements, and model inference have different authority. Identity and safety facts require more conservative promotion. The system must preserve child agency, avoid over-interpreting emotional moments, prevent attachment-maximizing behavior, and give parents appropriate transparency without turning every interaction into surveillance.
These differences belong in policy, admission, promotion, and context-compilation rules. The underlying event, evidence, temporal-belief, and procedure infrastructure can remain shared.
What the current ecosystem already provides
We do not need to invent every memory primitive. Several systems already demonstrate useful parts of this architecture, although they choose different boundaries.
| System | Strongest primitive | What it contributes to this design |
|---|---|---|
| Mem0 | Turnkey extraction, persistence, and semantic retrieval | A practical memory service that can sit behind an agent framework. Its built-in graph memory uses entity co-occurrence to improve retrieval ranking; it does not emit a separate relationship payload or create typed relationship edges. |
| Honcho | A continually updated representation of the user | Its peer model turns messages into premises and conclusions; its Get Context and Chat endpoints—the latter formerly called the Dialectic API—let an agent retrieve or ask questions of that model. Honcho also supports perspective-specific representations rather than assuming one universal truth. |
| Graphiti and Zep | Temporal knowledge graphs | Entities, relationships, episodes, provenance, and validity over time make them useful for evolving beliefs and relationship-heavy recall. |
| OpenClaw | Explicit memory tiers and disciplined curation | Inspectable files, provenance tracking via origin classes, deterministic write gates, background curation, standing intents, and recall-loop prevention offer strong operational patterns. |
| Hermes | A tiny always-present core plus searchable history | Bounded USER.md and MEMORY.md files keep stable context inexpensive, while SQLite full-text search makes detailed session history available on demand. |
| Letta | Agent-controlled state and memory blocks | Letta—formerly the project introduced as MemGPT—is a stateful agent runtime in which memory is part of the agent's execution model, useful when adopting the surrounding runtime as well as its memory abstractions. |
These approaches are complementary. Mem0 is close to a convenient memory API. Honcho is closest to an evolving user model and exposes that model as something the agent can reason with. Graphiti and Zep specialize in temporal relationships. OpenClaw contributes particularly good trust, promotion, and curation rules. Hermes demonstrates why a compact core and deep searchable history should remain separate. Letta shows the advantages of making memory a first-class part of a stateful agent runtime.
Honcho's current architecture illustrates a useful scaling pattern without batching derivation itself. Each message is synchronously stored and queued, then an asynchronous Deriver refines conclusions per message; a periodic Dreamer later consolidates and deepens those conclusions. The write path therefore stays cheap while understanding and longer-horizon reflection happen outside the caller's request. Its v3 Get Context feature also accepts a token budget, reinforcing the idea that context should be compiled for a turn rather than retrieved without bounds.
Our thesis is that a personal agent still needs a common substrate beneath these capabilities. Existing systems cover overlapping parts—and some reasonably describe themselves as memory infrastructure—but no single product should be assumed to own evidence, action audit, consent, source lineage, corrections, deletion, procedural learning, and child-specific policy for every application. Antara and HeyAppu.ai need those concerns to participate in one coherent lifecycle. No extraction library, graph, or agent runtime should become the sole canonical record by accident.
The pragmatic boundary is therefore:
- Adopt or integrate extraction, embeddings, hybrid search, graph projection, representation building, and reranking where an existing system performs well.
- Own the authoritative evidence and action journal, identity and tenancy, consent and source boundaries, belief and procedure lineage, lifecycle and deletion semantics, and the policy-aware context compiler.
- Keep projections replaceable. A Mem0 collection, Honcho representation, temporal graph, vector index, or curated memory file should be re-derivable under a recorded pipeline version—or explicitly treated as a cache—rather than becoming hidden, irrecoverable truth. Re-derivation may change inferred beliefs, so differences must be observable and corrected beliefs must remain anchored by journaled evidence.
The infrastructure shape
The memory platform is better understood as an event-driven system than as a database feature.
The journal requires per-subject ordering, idempotency, replay, and immutable lineage. Large or sensitive source artifacts belong in encrypted object storage. Asynchronous workers perform extraction, entity resolution, evidence aggregation, contradiction handling, embeddings, consolidation, decay, and procedure induction. Materialized views support current understanding, temporal history, semantic retrieval, relationships, action audit, and source-scoped deletion.
CockroachDB is a plausible authoritative substrate because transactional state, temporal metadata, evidence links, user isolation, job coordination, and vectors can coexist. Object storage handles large encrypted artifacts; KMS manages envelope encryption; a queue or change stream drives background computation. A graph index is optional and should be introduced only when relationship and temporal queries demonstrate that simpler projections are insufficient.
From one person’s memory to a system at scale
Everything above describes one person’s memory. Scale changes the problem along several independent dimensions: memory per person, connected sources, concurrent tasks, number of people, and eventually multiple agents acting for the same person or even one task. The most important consequence is that “memory” splits into scopes. Personal memory belongs to the user; task state belongs to a task; agent scratch space expires; the action journal is durable; and procedures are promoted rather than inherited automatically. A coordinator compiles a minimal packet for each specialist instead of copying the person’s context to every agent.
Part 2: What Personal-Agent Memory Looks Like at Scale develops the partitioning, multi-agent coordination, consistency, geographic placement, and reasoning-budget implications.
How the agent should behave
The agent should not directly rewrite its own identity model. It can emit typed evidence and propose interpretations; deterministic services and versioned policies decide how these affect durable memory.
When using memory, the agent should distinguish explicit fact from inference, tolerate missing or stale context, respect source and subject boundaries, and cite evidence internally for consequential decisions. Its outcomes and corrections flow back into the journal. Memory should influence behavior without becoming unquestionable truth.
This creates a productive loop:
The research lineage—and the opportunity
The roots are older than the current wave of agent-memory systems. The episodic/semantic distinction comes from cognitive psychology; cognitive architectures such as Soar and ACT-R implemented distinct memory systems decades ago; evidence-backed revisable beliefs go back to truth-maintenance systems; and “durable events, re-derivable views” is event sourcing applied to cognition.
This architecture combines several established ideas. CoALA separates working, episodic, semantic, and procedural memory. Generative Agents explores observation, reflection, and retrieval using relevance, recency, and importance. MemGPT—the project now known as Letta—treats context as a hierarchy that must be actively managed. Agent Workflow Memory learns reusable workflows from action trajectories. The Zep paper contributes temporal synthesis across conversational and business data, while the Graphiti implementation makes episode provenance explicit.
What remains underexplored is their combination as production infrastructure: a multi-tenant, event-sourced cognitive data plane where evidence and actions are durable, understanding and procedures are versioned and re-derivable, and each agent turn receives a policy-aware compilation of relevant context. In this design, consent, deletion, source boundaries, and per-product constitutions—including child-context safeguards—are stages of the memory lifecycle, not filters bolted on afterward.
HeyAppu.ai runs an early version of this architecture today; Antara is where we are pushing it toward many sources, broad personal context, consequential actions, and reusable operational knowledge. Part 2 covers what changes when memory, users, tasks, and agents scale along different dimensions.
The shared destination is not an assistant that remembers everything. It is an assistant that develops a disciplined understanding: selective about evidence, honest about uncertainty, capable of revising itself, and increasingly effective because it can learn both who it is helping and how to help them.
If you are building memory infrastructure for agents—or have encountered a failure mode this design would not survive—I would genuinely like to hear about it. And if parts of this substrate should exist as open infrastructure rather than inside individual products, that is a conversation worth having.