Working, Episodic, Semantic, and Procedural Memory

Memory is persisted state that may influence a later decision. That influence makes memory part of product behavior, security, and data governance—not a convenience wrapper around a vector database.

The phrase “agent memory” often collapses four different jobs:

working memory holds state for one active run;
episodic memory records events and outcomes associated with a principal;
semantic memory records durable propositions about a principal or workspace;
procedural memory records a named strategy or operating procedure.

Those records should not share one accidental lifetime or scope. A failed hypothesis from the current run must not become a permanent user fact. One user’s preference must not become a workspace procedure. An old successful trajectory must not silently override a newer policy. A summary must not make its source evidence impossible to recover.

This chapter builds mosaic-harness::memory_scope, a deterministic read boundary. Chapter 8 builds MP-22, the separate write gate for evidence support, conflicts, consent, retention, and expiration. Keeping read and write contracts separate prevents the convenient but unsafe design in which a model can both propose and authorize its own persistent belief.

source evidence and run state
            │
       Chapter 8 write gate
            │
      typed memory records
            │
 tenant + policy + principal + workspace + run + time
            │
      Chapter 7 read selector
            │
  selected memory receipt
            │
 untrusted/model-generated memory candidates
            │
       MP-21 context allocator

Learning objectives

By the end you will be able to:

distinguish working, episodic, semantic, and procedural memory by job and subject;
explain why a transcript, summary, cache, knowledge base, and model weights are not interchangeable;
bind every record to a tenant, policy, subject, writer, source set, and lifetime;
preserve original sources and known omissions for lossy summaries;
represent contested and revoked state separately from expiry;
use half-open time semantics so boundary behavior is unambiguous;
select memory using current authorization context rather than similarity alone;
prevent run-, principal-, and workspace-scoped records from crossing subjects;
record a decision for every selected and dropped item;
reproduce memory selection and detect stored-receipt mutation;
hand recalled records to context allocation without granting instruction authority;
diagnose memory omission, pollution, poisoning, contradiction, and stale-policy failures;
evaluate memory retrieval separately from final answer quality.

Prerequisites and dependency order

Complete behavior specifications, run envelopes, prompt releases, retrieval contracts, and context allocation first. The run envelope supplies the current tenant, principal, policy, workspace, clock, and run identity. Retrieval concepts apply to large memory stores, but authorization and typed scope must narrow the eligible corpus before semantic ranking. The context allocator decides whether selected memory is valuable enough to occupy the finite model input.

Then complete Chapter 8. It decides whether a candidate may become durable memory in the first place. This chapter deliberately assumes a record already passed that gate; it still rechecks read-time policy, subject, status, and expiry because authority and time can change after writing.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo test -p mosaic-harness memory_scope
cargo test -p mosaic-harness --example scoped_memory_selection
cargo run -q -p mosaic-harness --example scoped_memory_selection

The reference executes sixteen module tests and one complete example test. The fixture contains eight records. Four are selected:

working-current
semantic-user
procedure-build
episode-user

Four are dropped: working state from another run, semantic state for another principal, an expired episode, and a revoked procedure. The pinned identities are:

read context SHA-256    45e8bc0059ef0894f2e987de60a73edc35f14e9205f39983c148f0d82a520ea1
record set SHA-256      52a64ee56f383080df2ff25eda8d9b6bcebc50637df3eacbae632fe4fc6cabda
receipt SHA-256         be43e662b560f81f6c724591653c66bf193865e33052b2dd77bf1df15942380a

Memory is state, retrieval is an operation

A vector index is not memory by itself. It is one possible access path over records. The durable behavioral object is the record plus its validity rules:

pub struct MemoryRecord {
    pub schema_version: u32,
    pub record_id: String,
    pub kind: MemoryKind,
    pub tenant_id: String,
    pub policy_sha256: String,
    pub subject: MemorySubject,
    pub status: MemoryStatus,
    pub content_sha256: String,
    pub provenance_sha256: String,
    pub token_count: u32,
    pub token_counter_release: String,
    pub created_at_unix_ms: u64,
    pub expires_at_unix_ms: Option<u64>,
    pub writer_release: String,
    pub sources: Vec<MemorySource>,
    pub summary: Option<SummaryDescriptor>,
}

The index can be rebuilt. A record whose subject, evidence, status, or lifetime was never stored cannot be repaired by a better embedding.

MemGPT describes virtual context management: moving information among memory tiers to work beyond a finite model context. That is a useful systems analogy. It does not make external memory infinite or automatically correct. Moving bytes from durable storage into a prompt is a policy decision with cost, authority, and failure modes.

Give each memory kind one job

Working memory

Working memory is state for one run:

selected plan and intermediate decisions;
observed tool results;
unresolved hypotheses;
partial artifacts;
checkpoints and progress markers;
facts needed by a later step in the same run.

The reference requires a Run subject and exact current run_id. Working state from run-old is not eligible in run-current. If a workflow resumes the same durable run, it may read that run’s working state after verifying the run generation and policy. A new retry with a new identity should not inherit it accidentally.

Working state can contain unverified hypotheses. Store epistemic status explicitly. The reference selector admits only records whose write gate marked them verified, but a fuller execution-state store may retain provisional items outside the prompt so an investigator can test them.

Working memory is not a substitute for a trace. A trace records the causal history of what happened. Working memory records the state needed to continue. Derive state from events where possible and keep both identities connected.

Episodic memory

Episodic memory represents an event or outcome:

“the user rejected a verbose answer in run R”;
“this repair strategy passed the verifier”;
“a deployment failed because migration M was missing”;
“the tool returned permission denied under policy P.”

An episode should preserve when, where, and under which conditions it occurred. It is evidence about an experience, not a universal rule. A successful strategy under one repository, model, or policy may fail elsewhere.

The reference scopes episodes to a principal. A production record will often add workspace, task family, environment, model release, and behavior-contract identity. Retrieval should prefer episodes whose preconditions match the current task, not merely those whose prose sounds similar.

Do not turn model self-critique into verified episodic truth without observable outcome evidence. “I failed because the prompt was ambiguous” is a hypothesis. The verifier result, tool receipt, and trace are evidence.

Semantic memory

Semantic memory represents a proposition intended to remain useful:

a user preference with consent;
a workspace’s chosen architecture;
a verified domain fact;
a durable entity or relationship;
an accepted decision and its rationale.

The subject matters. A principal preference such as output language should not become a workspace fact. A workspace decision should not follow the same person into another workspace.

Semantic records need conflict handling. Two records can disagree without either being malformed. Chapter 8 defines supersession, contesting, and adjudication. The read path in this chapter excludes contested records rather than arbitrarily choosing the newest or most confident one.

Confidence is metadata, not proof. A model-generated probability does not replace sources, verification, or an accountable write decision.

Procedural memory

Procedural memory represents how to act:

a repository build procedure;
a debugging sequence;
a tool-use strategy;
a response or review protocol;
a learned decomposition for a task family.

This category is security-sensitive because procedural text resembles instructions. The reference uses a Procedure { workspace_id, procedure_id } subject and still converts selected records to ContextTrust::ModelGenerated. Persistence does not promote a procedure to system or developer authority.

A procedure should bind:

applicable workspace and task family;
required capabilities and tools;
policy and behavior-contract compatibility;
source trajectories and verification results;
writer/optimizer release;
held-out evaluation result;
expiry or review schedule;
superseded and rollback identities.

Treat procedural-memory changes like program changes. Evaluate and release them; do not let one successful trajectory rewrite production behavior live.

Keep subject shape in the type system

The reference subject enum is:

pub enum MemorySubject {
    Run { run_id: String },
    Principal { principal_id: String },
    Workspace { workspace_id: String },
    Procedure {
        workspace_id: String,
        procedure_id: String,
    },
}

Validation permits only these combinations:

working     → run
episodic    → principal
semantic    → principal or workspace
procedural  → procedure within workspace

This removes invalid states before selection. A Working + Principal record is rejected rather than interpreted differently by two callers.

Real products may need team, organization, device, conversation, project, jurisdiction, or data-classification scopes. Add explicit variants. Do not solve every scope with nullable columns and an undocumented precedence rule.

Bind tenant and policy twice

Every record carries tenant_id and policy_sha256. Every read context carries the current values. Both must match.

Tenant binding prevents direct cross-customer recall. Policy binding prevents a record written under one permission/retention contract from being assumed valid under another. A policy change may:

revoke the source data;
disallow personalization;
shorten retention;
change which principals may read workspace state;
require re-verification;
change redaction or residency rules.

Policy mismatch causes a typed drop. A production system may run a migration or re-authorization job and issue a new record; it should not edit the old record in place and erase history.

Cache keys must include tenant, current policy, record-set or index snapshot, selector release, and read subject. Semantic similarity is never a safe cache identity by itself.

Model time explicitly

The record uses:

created_at_unix_ms: u64
expires_at_unix_ms: Option<u64>

An expiry must be later than creation. Eligibility is half-open:

created_at <= now < expires_at

At now == expires_at, the record is expired. This avoids two services disagreeing at the boundary. Use a named, injected clock in production tests. Persist the clock domain if records can be produced from external event time rather than service wall time.

Expiry and revocation are different:

expiry is a predeclared time boundary;
revocation is an explicit decision that the record must no longer influence behavior;
contesting says the proposition is unresolved or contradicted;
supersession links an old valid record to a replacement.

Deleting the only record can erase audit evidence. Depending on privacy and retention obligations, store a minimal tombstone with identity, reason, time, and authorization while deleting the sensitive content and derived indexes.

Make summaries source-restorable

A summary is a lossy cache. It may omit the detail that becomes decisive later. The reference SummaryDescriptor records:

pub struct SummaryDescriptor {
    pub objective: String,
    pub summarizer_release: String,
    pub source_set_sha256: String,
    pub known_omissions: Vec<String>,
}

Every record also contains one or more source identities with content, provenance, and locator digests. The selector therefore never returns an anonymous paragraph whose origin is unknowable.

The ReadAgent research is a useful pattern: compressed “gist” memories can support long-context work while the system retains an operation to return to original passages. The production lesson is not that one summary prompt is universally sufficient. It is that compression and source lookup are separate operations and should both be evaluated.

Record the summary objective because “summarize for incident chronology” and “summarize user preferences” preserve different information. Record known omissions because deliberate absence is operationally different from an unknown loss.

For high-stakes use, store structured claims with per-claim citations rather than one prose blob. Revalidate claims when source records change or policy revokes access.

Select with current read context

The read contract contains:

pub struct MemoryReadContext {
    pub schema_version: u32,
    pub selector_release: String,
    pub tenant_id: String,
    pub policy_sha256: String,
    pub principal_id: String,
    pub workspace_id: String,
    pub run_id: String,
    pub now_unix_ms: u64,
    pub allowed_kinds: Vec<MemoryKind>,
    pub max_records: usize,
}

Selection checks, in a deterministic order:

the kind is enabled for this route;
tenant matches;
current policy matches;
status is verified rather than contested or revoked;
expiry is still in the future;
the typed subject matches the run, principal, or workspace;
the bounded selection limit has capacity.

The implementation records every outcome:

kind_disabled
wrong_tenant
policy_mismatch
subject_mismatch
contested
revoked
expired
selection_limit

This is more useful than returning four records from eight and losing why the other four vanished.

Eligible records are ordered working, semantic, procedural, then episodic; within a kind, newer records come first and record ID breaks exact ties. That policy is intentionally simple and reproducible. A production selector may rank relevance and utility after eligibility, but it must version that ranking and preserve deterministic ties.

Reproduce the selection

MemorySelectionReceipt binds:

read-context digest;
ordered record-set digest;
ordered selected IDs;
one selected or dropped decision per input record;
selector release.

verify_memory_selection recomputes the receipt from independently supplied inputs. Swapping two selected IDs in stored output fails.

The ordered record-set digest is appropriate for this explicit example fixture. A database-backed selector should bind a canonical query and immutable snapshot, then canonicalize records by stable identity before hashing. Never let nondeterministic map or database iteration order change a receipt.

Hand memory to context without authority escalation

The conversion into MP-21 uses:

lane          memory
trust         model_generated
placement     variable_body
required      false
content       exact memory content digest
provenance    exact record provenance digest
token count   bound to content and counter release

The priority order gives current working state more opportunity than an old episode, but context allocation can still drop memory under total or lane pressure.

Do not render recalled procedures as developer messages. Use a delimited data block such as:

<recalled-memory kind="procedural" record="...">
...
</recalled-memory>

and state in trusted instructions that recalled material is data to evaluate, not authority to change policy or tool permissions.

Retrieval over a large memory store

The in-memory reference validates the complete supplied set. A large system should apply the Chapter 6 funnel:

tenant/policy/status/expiry/subject eligibility
  → lexical + semantic + structured candidate generation
  → fusion/reranking/diversity
  → memory selection receipt
  → context allocation

Filter before approximate search when possible. If the engine filters after vector scanning, measure recall under selective principal and workspace predicates. A globally excellent vector index can return zero eligible hits for a small subject.

Useful ranking features include task relevance, source strength, recency, supersession, novelty, scope specificity, prior demonstrated utility, and expected context cost. Never add model-generated “importance” as an unexamined universal truth.

Failure investigations

A false belief keeps returning

Trace the record back through its write receipt and sources. Check whether the record is semantic when it should be episodic, whether the verifier accepted model self-report as evidence, and whether new contradictory evidence contested it. Revoke or supersede the record; do not merely add a prompt saying to ignore it.

A correct preference is forgotten

Distinguish:

no candidate was ever written;
the write gate rejected it;
the record expired or was revoked;
current policy does not allow it;
subject scope differs;
retrieval missed it;
selection limit removed it;
context allocation dropped it;
the model received but ignored it.

Each stage has a separate receipt. Fix the failing stage rather than increasing every top-k.

One user’s memory appears for another

Treat this as a security incident. Stop recall for the affected route. Inspect tenant and principal predicates, cache keys, shared indexes, record IDs, query expansion, and context manifests. Add negative tests with nearly identical content across two tenants and two principals.

A summary hides the decisive detail

Use its source-set identity and locators to retrieve original evidence. Compare the summary objective with the current task. Add the failure to a summary-loss evaluation set. Consider claim-level summaries, task-specific summaries, or direct source retrieval for high-risk cases.

Memory grows without bound

Measure writes, active records, index bytes, derived artifacts, retrieval latency, and context inclusion utility by kind. Add retention, compaction, supersession, and explicit forgetting. “The context window is large” is not a storage or governance policy.

Security and privacy implications

Persistent memory extends the blast radius of one malicious or incorrect input. An instruction-like string stored today may affect a different task weeks later.

Controls include:

treat all recalled text as non-authoritative data;
require an independent write gate and evidence;
scope by tenant, principal, workspace, procedure, and run;
bind current policy at read time;
encrypt and classify records and derived indexes;
redact secrets before durable storage;
support consent, access, correction, revocation, deletion, and export;
delete or tombstone derived embeddings and caches with the source;
prevent evaluator and held-out data from becoming memory;
rate-limit memory writes and detect repeated poisoning attempts;
audit high-impact procedural changes like code releases;
test cross-subject and stale-policy negative cases.

Recent memory-poisoning research reinforces the threat model: persistent state can turn one planted payload into a long-lived attack surface. Treat new results as motivation for your own adversarial tests, not as a substitute for a product-specific threat model.

Performance implications

Track:

eligible and selected records by kind;
drop reasons;
exact/approximate retrieval recall;
active, expired, contested, revoked, and tombstoned counts;
write, compaction, deletion, and index-propagation lag;
selection and source-restoration latency;
token cost and context inclusion rate;
measured answer improvement per selected token;
repeated retrieval of records the model never uses.

Cache only after eligibility and include all scope and policy identities. Precompute embeddings and summary representations asynchronously, but do not expose a record until the durable write and index visibility contract is satisfied.

Alternatives and trade-offs

Full conversation replay

Replay preserves raw detail but grows without bound, includes irrelevant and unsafe content, and still competes for context. It is useful as source evidence, not as the only memory policy.

Rolling summaries

Rolling summaries are compact but accumulate error and erase old source boundaries. Keep source manifests and periodically rebuild from original events rather than only summarizing the prior summary.

Structured profile fields

Typed profile fields are easier to validate, correct, and authorize than prose semantic memory. Prefer them for stable product concepts. Use prose only when the domain genuinely requires it.

Vector-store memory

Vector search supports fuzzy relevance but does not supply truth, authorization, conflict resolution, retention, or authority. Place it behind the typed record and read contracts.

Fine-tuning or model weights

Weights are not an appropriate mutable store for per-user facts. They are difficult to inspect, delete, scope, correct, and cite. Fine-tuning belongs to the later adaptation section with dataset governance and release gates.

Exercises

Recall

Why is a successful episode not automatically a procedure?
Why do records bind both write-time and current read-time policy?
What is the difference between contested, revoked, expired, and superseded?
Why must a summary retain source identities and an objective?
Why is verified memory still not developer authority?
Which receipt proves memory selection, and which manifest proves prompt inclusion?

Implementation

Add Superseded { replacement_id } and reject cycles.
Add workspace scope to episodic records without allowing cross-workspace recall.
Add a typed, canonical relevance-stage receipt after eligibility.
Add a signed source-set manifest and verify SummaryDescriptor.source_set_sha256.
Add property tests for deterministic ordering under shuffled record input.
Add tombstones and prove a deleted record cannot remain in the retrieval index.
Join memory selection to allocate_context and report which selected memories were later dropped.
Add a manual clock and boundary tests for expiry, policy validity, and revocation time.

Failure laboratory

Mutate tenant, principal, workspace, run, policy, status, expiry, content digest, source locator, summary objective, ordering, and receipt selection. Every authorization or identity mutation should fail or produce the exact typed drop. Store an instruction-like payload as content and prove it remains a model-generated memory block rather than a trusted instruction.

Design

Design memory for a Rust coding assistant used across personal and company repositories. Specify:

record kinds and subjects;
consent and write authority;
source evidence and verification;
conflict and supersession;
retention and deletion;
retrieval and ranking;
context allocation;
procedural release gates;
security tests;
evaluation metrics;
user inspection and correction.

Explain how the system behaves when a user leaves a company workspace but keeps a personal account.

Solution guidance

Model supersession as an immutable edge from old to new. Reject self-reference, unknown replacement, cross-subject replacement, cycles, and replacement by an older policy. Keep the old record status for audit while excluding it from normal recall.

For deterministic relevance, use fixed-point scores and stable record-ID ties. Hash the complete query, eligible-set snapshot, feature release, ranker release, and ordered results.

Deletion tests should inspect the source store, semantic index, lexical index, caches, summaries, and context candidates. A database delete alone is not complete.

The prompt-injection test should pass storage validation as data when its sources are legitimate. Security comes from authority separation, write verification, scoped retrieval, and safe rendering, not from pretending suspicious text never exists.

Check your understanding

Can working memory be reused by a new run?
Can episodic memory directly grant tool permissions?
When is a principal semantic record eligible?
What happens at now == expires_at?
Does deleting a vector remove the underlying memory?
Can two verified semantic records contradict each other?
Why does policy mismatch drop an otherwise relevant record?
How can summary loss be repaired during a run?
Which scope should own a build procedure?
What evidence would justify promoting an episode into a procedure?

Completion checklist

Authoritative references

Packer et al., “MemGPT: Towards LLMs as Operating Systems”
Park et al., “Generative Agents: Interactive Simulacra of Human Behavior”
Google DeepMind, “A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts”
LangChain documentation, semantic, episodic, and procedural memory concepts
Retrieval Contracts, Snapshots, and Reranking
Context Manifests and Budget Allocation
mosaic-harness::memory_scope, typed scope and deterministic read-selection implementation