AI Agent Memory: Schema-Guided State and Provenance

Viacheslav Dubrov

2026-06-20 · Updated 2026-07-22 · 14 min · AI Agents

Long-running agents often retrieve stale facts because ordinary semantic memory has no rule for deciding which value is current.

When a user changes a passport deadline from July 15 to June 30, vector search can retrieve both statements. A stateful memory layer must record that the June 30 value supersedes the earlier one.

For engineers building long-running or multi-tenant agents, that stale-fact failure is why memory must survive separate runs while enforcing current state and tenant boundaries. The design below shows how to write typed records and test current versus point-in-time reads without treating vector recall as the source of truth.

The context-window trap

The context window is the input available to one model call. An application may carry messages into later calls, but application policy must decide which old facts remain true and who may see them.

Long-running agents need to remember preferences, task status, customer facts, tool decisions, compliance notes, and prior mistakes. The easy version is to append summaries or dump old notes into a vector store. That works until one of the remembered facts changes.

Now the agent has two passport deadlines, two preferred formats, or two project decisions. Semantic search may retrieve both. A summary may overwrite one. A long context may include the stale one next to the active one. These designs recall text while leaving the current value unenforced.

The memory contract has to answer concrete questions:

• What is true now?
• What was true on June 2?
• Who said it?
• Which tenant does it belong to?
• Which older fact did this fact replace?
• Can I delete or expire it?

Schema-Guided Agent Memory (SGAM) stores those answers as fields and relations instead of leaving them implicit in prose.

What SGAM means

Three similarly named ideas establish the scope of SGAM.

Schema-Guided Dialogue (SGD) is the 2019 Google task-oriented dialogue dataset. Its schema describes service APIs, intents, and slots so a dialogue model can track state for services it has not seen before. It is useful precedent for schema-based tracking, with a scope limited to dialogue services.

Schema-Guided Memory (SGM) is the research term used by Mei et al. in According to Me: Long-Term Personalized Referential Memory QA. The paper compares free-text Descriptive Memory (DM) with fixed-schema key-value memory items. Both representations contain the same source information in different structures.

In this article, I use Schema-Guided Agent Memory (SGAM) to mean an engineering pattern in which schemas govern writes, updates, retrieval, and deletion. The schema defines application state and its lifecycle.

ATM-Bench shows why the representation matters. It uses roughly four years of personal data from emails, images, and videos. The questions require personal references, location, multiple pieces of evidence, and updates over time. On its hard split, the paper reports that SGM outperforms DM for retrieval and question answering under the tested setup. SGM exposes fields such as time, source, location, entities, and tags in a fixed representation. I treat that representation and the reported results as the supported conclusion; the paper does not establish a direct field-addressing mechanism.

SGM versus DM answers a storage question: should memory remain free text, or should it use named fields? A production agent has another problem before storage. It must turn an unstructured conversation into a proposed memory update. Schema-Guided Reasoning (SGR) makes that intended decision path inspectable: the response can include evidence, the subject and attribute, a comparison with current state, and a candidate write. The model response does not enforce dependencies or order between those fields. Separate calls, validators, and application policy enforce those rules; SGAM applies the storage and lifecycle rules after the model call.

Separate model extraction from memory ownership

The memory write crosses three layers. Structured Output (SO) enforces the candidate object’s shape. Schema-Guided Reasoning (SGR) makes the model’s intended fields and decision intent inspectable in a structured response. Schema-Guided Agent Memory (SGAM) manages the candidate as durable state after the model call. Separate calls, validators, and application policy enforce dependencies, order, and lifecycle rules.

A verdict, route, or plan usually expires with the current request. Another run may read a memory candidate days later or use it to choose a tool call. That longer lifetime demands storage rules that SGR doesn’t provide.

SGR structures one model call around an intended reasoning topology. For a memory write, the response might include source evidence, a normalized subject and attribute, a comparison with current state, and a proposed update. Pydantic or JSON Schema describes those fields. Provider-native Structured Output or a guided decoding runtime such as XGrammar keeps the response in that shape.

That shape makes the model’s declared fields and intent inspectable. It cannot guarantee a correct conclusion or prove that the model used those fields in order. Separate calls, validators, and application policy are the enforcement boundary for dependencies, order, and lifecycle.

SGAM decides what happens after that object exists. Should it be stored? Does it supersede an older fact? Which tenant can see it? Is it current or historical? Which source episode backs it?

The table sets out the ownership and failure mode of each layer:

On the write path, the sequence is:

SGR reasoning schema + Structured Output -> candidate write -> SGAM policy and persistence

The following illustrative excerpt from memory_models.py defines the object passed from extraction to the SGAM write service. This block needs Pydantic, so it is marked no-run in the example checks for environments that do not install that optional dependency:

from datetime import datetime
from pydantic import BaseModel, Field

class MemoryDelta(BaseModel):
    tenant_id: str = Field(description="Isolation boundary, e.g. acme")
    subject: str = Field(description="Normalized entity ID, e.g. mira")
    attribute: str = Field(description="Property being updated")
    value: str = Field(description="New value")
    valid_from: datetime
    source_episode_id: str

MemoryDelta captures what the model extracted. The SGAM write service still decides whether to reject, merge, or store it.

Write and read paths have different jobs

Only the write path mutates stored state. The read path selects records for the current request.

The ingestion flow is the write path:

1. Capture a raw episode from messages, tool results, or business events.
2. Extract typed candidates through structured output.
3. Validate the schema and reject malformed writes.
4. Reconcile conflicts, close stale facts, and keep provenance.
5. Commit the record to the SGAM store.

The request flow is the read path:

1. Start with the user question.
2. Decide whether the question needs current state or point-in-time state.
3. Filter by tenant, memory type, subject, attribute, and validity window.
4. Add vector or graph expansion only if the exact state lookup is not enough.
5. Assemble the smallest cited context for the model.

Read that diagram from left to right in two lanes. The top lane writes memory, and the bottom lane reads it. Both use the same store.

What belongs in a memory schema

A minimal SGAM record needs more than text.

tenant_id
memory_id
subject
attribute
value
memory_type
schema_version
valid_from
valid_to
supersedes_memory_id
source_episode_id
confidence
retention_policy

With these fields, a new passport deadline can close the previous deadline without erasing history. The same table can answer current and point-in-time queries, then trace the result to its source episode. schema_version supports migrations, while retention_policy tells deletion jobs what else they must remove.

Use RAG to retrieve documents and SGAM to maintain mutable state. Vector search still belongs in the system for fuzzy recall, clustering, and expansion. The current value of mira.passport_deadline should come from a scoped memory record rather than whichever chunk happened to rank first.

A stale-fact example

Consider a synthetic two-episode trace represented as a DM baseline and an SGAM ledger:

e1: Mira prefers concise answers. Her passport deadline is 2026-07-15.
e2: Mira corrected the deadline. It is now 2026-06-30.

A DM baseline keeps both episodes as free text, so text search can return e1 because it contains the right words. SGAM extracts a typed fact from each episode, keyed by tenant, subject, and attribute. It should return e2 as current state and keep e1 for a historical query.

The following is a self-contained SQLite write path. It uses ISO-8601 UTC strings for timestamps, configures sqlite3.Row before reading by column name, and represents facts as half-open intervals: [valid_from, valid_to). The table rejects invalid intervals and has a partial unique index for one open fact per tenant, subject, and attribute. replace_fact owns one transaction; SQLite serializes concurrent writers, so callers should retry the whole transaction after a lock or uniqueness error.

import sqlite3
from dataclasses import dataclass

@dataclass(frozen=True)
class MemoryFact:
    tenant_id: str
    subject: str
    attribute: str
    value: str
    valid_from: str
    source_episode_id: str

def open_db() -> sqlite3.Connection:
    db = sqlite3.connect(":memory:")
    db.row_factory = sqlite3.Row
    db.executescript(
        """
        create table memory_facts (
            fact_id integer primary key,
            tenant_id text not null,
            subject text not null,
            attribute text not null,
            value text not null,
            valid_from text not null,
            valid_to text,
            source_episode_id text not null,
            check (valid_to is null or valid_from < valid_to)
        );
        create unique index one_open_fact
        on memory_facts (tenant_id, subject, attribute)
        where valid_to is null;
        """
    )
    return db

def replace_fact(db: sqlite3.Connection, fact: MemoryFact) -> None:
    with db:
        exact = db.execute(
            """
            select fact_id
            from memory_facts
            where tenant_id = ? and subject = ? and attribute = ? and valid_from = ?
            """,
            (fact.tenant_id, fact.subject, fact.attribute, fact.valid_from),
        ).fetchone()
        if exact:
            raise ValueError("equal valid_from requires an application conflict policy")

        containing = db.execute(
            """
            select fact_id, valid_to
            from memory_facts
            where tenant_id = ?
              and subject = ?
              and attribute = ?
              and valid_from < ?
              and (valid_to is null or valid_to > ?)
            limit 1
            """,
            (
                fact.tenant_id,
                fact.subject,
                fact.attribute,
                fact.valid_from,
                fact.valid_from,
            ),
        ).fetchone()
        if containing:
            db.execute(
                "update memory_facts set valid_to = ? where fact_id = ?",
                (fact.valid_from, containing["fact_id"]),
            )
            successor_boundary = containing["valid_to"]
        else:
            successor = db.execute(
                """
                select valid_from
                from memory_facts
                where tenant_id = ? and subject = ? and attribute = ? and valid_from > ?
                order by valid_from
                limit 1
                """,
                (fact.tenant_id, fact.subject, fact.attribute, fact.valid_from),
            ).fetchone()
            successor_boundary = successor["valid_from"] if successor else None

        db.execute(
            """
            insert into memory_facts
                (tenant_id, subject, attribute, value, valid_from, valid_to, source_episode_id)
            values (?, ?, ?, ?, ?, ?, ?)
            """,
            (
                fact.tenant_id,
                fact.subject,
                fact.attribute,
                fact.value,
                fact.valid_from,
                successor_boundary,
                fact.source_episode_id,
            ),
        )

def fact_at(db: sqlite3.Connection, timestamp: str) -> sqlite3.Row:
    return db.execute(
        """
        select value, source_episode_id
        from memory_facts
        where tenant_id = ?
          and subject = ?
          and attribute = ?
          and valid_from <= ?
          and (valid_to is null or valid_to > ?)
        limit 1
        """,
        ("acme", "mira", "passport_deadline", timestamp, timestamp),
    ).fetchone()

db = open_db()
replace_fact(
    db,
    MemoryFact("acme", "mira", "passport_deadline", "2026-07-15", "2026-06-01T09:00:00Z", "e1"),
)
replace_fact(
    db,
    MemoryFact("acme", "mira", "passport_deadline", "2026-06-30", "2026-06-03T10:00:00Z", "e2"),
)
current = fact_at(db, "2026-06-04T00:00:00Z")
assert (current["value"], current["source_episode_id"]) == ("2026-06-30", "e2")

historical = fact_at(db, "2026-06-02T00:00:00Z")
assert (historical["value"], historical["source_episode_id"]) == ("2026-07-15", "e1")

# A delayed extraction predates e1. It ends at e1's existing boundary,
# rather than closing e2, which remains current.
replace_fact(
    db,
    MemoryFact("acme", "mira", "passport_deadline", "2026-08-01", "2026-05-30T08:00:00Z", "e0"),
)
delayed = fact_at(db, "2026-05-31T00:00:00Z")
assert (delayed["value"], delayed["source_episode_id"]) == ("2026-08-01", "e0")
current = fact_at(db, "2026-06-04T00:00:00Z")
assert (current["value"], current["source_episode_id"]) == ("2026-06-30", "e2")
intervals = db.execute("select valid_from, valid_to from memory_facts").fetchall()
assert all(row["valid_to"] is None or row["valid_from"] < row["valid_to"] for row in intervals)

episodes = (
    "e1: Mira prefers concise answers. Her passport deadline is 2026-07-15.",
    "e2: Mira corrected the deadline. It is now 2026-06-30.",
)
naive_match = next(episode for episode in episodes if "passport deadline" in episode)
assert naive_match.startswith("e1:")

replace_fact first rejects an equal valid_from: that conflict needs an application policy, such as source precedence or a user-confirmed revision, rather than a silent overwrite. Otherwise it finds the interval containing the incoming timestamp. If it finds one, it closes that predecessor at the incoming boundary and gives the inserted row the predecessor’s former end. If the timestamp is before every stored interval, it uses the next interval’s valid_from as the new row’s end. Thus a delayed fact before e1 becomes [2026-05-30, 2026-06-01) and leaves e1 and current e2 intact. The row factory makes row["fact_id"] and named result fields valid on the default connection created here. The partial unique index is the database guard for one open row; a multi-writer deployment should choose a database and retry policy suited to its workload. DM has no equivalent update step, so the old text can still outrank the correction.

The assertions verify the current e2 row, the historical e1 row, delayed insertion before e1, the interval invariant, and the first matching naive text episode:

Naive text memory:
  returned episode: e1 -> passport deadline is 2026-07-15

SGAM current state:
  mira.passport_deadline = 2026-06-30
  valid_from=2026-06-03T10:00:00Z, source=e2

SGAM point-in-time state:
  on 2026-06-02, mira.passport_deadline = 2026-07-15

In production, pair this transaction with structured extraction on the write path. The database transaction updates temporal validity. The model extracts a candidate fact but doesn’t decide which stored row remains current.

Storage choices follow the retrieval pattern

Projects use several names for parts of this pattern: memory stores, context graphs, profiles, long-term stores, graph RAG, and stateful agents.

Graphiti is a concrete open-source implementation of relational, temporal memory. It tracks facts as they change, keeps pointers to source episodes, and supports hybrid retrieval. LangGraph separates thread checkpoints from cross-thread stores. Mem0 packages memory operations as a managed service. Letta uses editable context blocks rather than field-level SGAM, but it still treats agent state as persistent data.

Start with the data model. If exact fact lookup is the main operation, a relational table with JSON payloads, validity columns, tenant indexes, and a vector sidecar is usually enough. Add a graph when relationship traversal is part of the product, not because the graph demo looks impressive.

Build the write path before the graph

First decide what the product is allowed to remember. The graph-versus-vector choice comes later.

A support agent might remember account tier, open cases, and durable contact preferences. It shouldn’t promote every frustrated aside into profile state. A coding agent might remember repo conventions and unresolved tasks. It shouldn’t keep a private note forever because that note happened to be retrieved once.

Start with the write path and treat memory as a small state mutation:

1. Name the memory type, subject, tenant scope, and retention class.
2. Extract candidate records with structured output.
3. Validate the payload with Pydantic or the schema layer your stack already uses.
4. Resolve conflicts before insert, including whether the new record supersedes an old one.
5. Keep a source pointer to the raw episode, tool result, file, ticket, or user confirmation that produced the record.
6. Write the schema version with each record instead of leaving it only in application code.

The first SGAM store can be a relational table with a JSON column and a few indexes. A graph becomes useful when the product needs to traverse relationships such as customer-to-account, account-to-policy, task-to-artifact, or project-to-decision.

Hot path and background writes

Immediate extraction is worth it when the next turn depends on the new memory. If the user says “remember that I prefer short answers,” the system should not need a nightly job before it behaves differently.

Most turns don’t need an immediate write. Save the raw episode with tenant, session, and tool metadata, then let a background worker extract candidates later. With recurrence-based consolidation, the worker buffers weak signals and promotes a fact only after similar evidence repeats or the user confirms it. This adds freshness lag. That is acceptable for “user often asks for CSV exports” and risky for “customer changed the delivery address.”

Keep the read path deterministic. Enforce tenant scope and validity first, then use fuzzy retrieval only when it can add useful context.

1. Filter by tenant, memory type, and validity window.
2. Retrieve exact structured state before semantic neighbors.
3. Use vector or graph expansion for supporting evidence, related entities, and examples, not as the authority for current facts.
4. Assemble the smallest cited context that can answer the question.

Treat schema migration as a product change because it alters what the agent can recall, cite, or delete. It can also change which historical facts count as current. Plan migration scripts, backfills, dual-read windows, and deletion behavior in the same release.

When SGAM is worth the complexity

Use SGAM when facts can change over time:

• user preferences that can be updated or revoked
• customer or account facts with audit requirements
• task state for long-running assistants
• coding-agent project memory
• multi-agent shared state
• compliance notes where provenance matters
• temporal questions such as “what did we believe before the migration?”

SGAM is overkill when memory is short-lived, exploratory, or cheap to recompute. If the agent only needs a few turns of continuity, a checkpoint and trimmed message history are enough. Static document QA may need only RAG. And if the domain is so unsettled that the schema changes every day, typed memory will slow the team down.

Evaluation checklist

Evaluate the memory lifecycle as well as the final answer. A system can produce a plausible response after it wrote the wrong fact, retrieved a stale one, or crossed a tenant boundary.

I use the same stage-by-stage split as my RAG evaluation article. Measure the stage where a failure can happen rather than limiting evaluation to the generated text. The trace discipline from the agent evaluation article also applies because a memory bug often appears in the run history before it reaches the answer.

I would test SGAM with replay. Feed a fixed sequence of episodes into the memory writer and inspect the ledger after each meaningful turn. Then ask current-state and point-in-time questions against the resulting store.

Benchmarks such as LoCoMo, LongMemEval, and ATM-Bench provide public test cases. They don’t replace a domain test suite. A coding assistant, customer support bot, and compliance copilot need different schemas, filters, retention rules, and failure tests.

Caveats

SGAM is my label for a pattern, not a standard. Existing projects divide the problem differently. LangGraph memory and LangMem describe short-term and long-term stores, profiles, collections, hot-path writes, and background memory managers. Zep Graphiti uses the term temporal Context Graph. Letta persists editable memory blocks, while Mem0 offers a managed memory layer. Microsoft GraphRAG, LlamaIndex property graphs, and Cognee frame related parts of the problem as knowledge graphs.

A user profile, an episode log, a document graph, and an agent-editable memory block solve different retrieval and update problems. I reserve SGAM for durable memory that represents current application state and therefore needs schema, validity, provenance, conflict handling, retention, and migration.

Typed memory can still be wrong. A schema makes bad writes easier to inspect; it does not make them trustworthy. You still need source trust, user confirmation for sensitive facts, conflict policy, deletion, and monitoring.

Schema migration is work. Once memory becomes state, you own versioning, backfills, old records, and deletion behavior. Skip that work and old records will outlive the semantics or retention policy that created them.

References

• According to Me: Long-Term Personalized Referential Memory QA - Mei et al. paper introducing ATM-Bench and Schema-Guided Memory.
• Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset - Rastogi et al. paper on the SGD dataset.
• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory - Wu et al. benchmark for long-term memory abilities.
• Graphiti: Build Temporal Context Graphs for AI Agents
• Zep: A Temporal Knowledge Graph Architecture for Agent Memory
• Mem0 Platform Overview
• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
• LangGraph Memory Overview
• LangGraph Persistence
• LangMem documentation
• Letta: Introduction to Stateful Agents
• Microsoft GraphRAG documentation
• LlamaIndex: Using a Property Graph Index
• Cognee Documentation
• Pydantic model validation docs
• Python sqlite3 documentation