Memory Stack

A five-layer memory architecture for long-running AI assistants that stays coherent after months of daily use — because each layer is allowed to fail without taking the agent down.

The core insight: no single memory system is reliable enough to be the only one. Vector stores drift. Summaries lose detail. Context windows overflow. The fix is not a smarter single system — it's five boring layers, each with one job, each able to die independently.

This is the architecture behind a production assistant that has stayed consistent across thousands of daily conversations. It's a design, not a library — you can build the useful 80% in an afternoon with three markdown files and a cron job.

The five layers

┌───────────────────────────────────────────────┐
│ 5. WRITE-BACK    after every successful turn   │
├───────────────────────────────────────────────┤
│ 4. RETRIEVAL     semantic recall over all of   │
│                  it (vector / graph lane)      │
├───────────────────────────────────────────────┤
│ 3. LONG-TERM     curated facts, decisions,     │
│                  preferences (one file)        │
├───────────────────────────────────────────────┤
│ 2. SHORT-TERM    rolling 3-5 day summary       │
│                  (one file, rewritten daily)   │
├───────────────────────────────────────────────┤
│ 1. RAW LOG       daily append-only logs        │
│                  (memory/YYYY-MM-DD.md)        │
└───────────────────────────────────────────────┘

Layer 1 — Raw daily logs (ground truth)

Every user message and agent reply appended to memory/YYYY-MM-DD.md. Append-only, never edited, never summarized in place. This is the layer you trust when the layers above disagree. Cost: almost nothing. Value: you can always reconstruct what actually happened.

Layer 2 — Short-term rolling summary

One file (SHORT_TERM.md) holding a 3-5 day summary: active projects, open loops, recent decisions. Rewritten daily by a scheduled job that reads the raw logs. Loaded into context every turn. This is what makes the agent feel "caught up" without loading a week of transcripts.

Layer 3 — Long-term curated facts

One file (LONG_TERM.md) of durable facts: who the user is, standing preferences, infrastructure decisions, things that must never be forgotten. Entries get promoted here from short-term when they prove durable. Relative dates become absolute dates on write. Pruned periodically — a long-term file that grows forever becomes noise.

Layer 4 — Semantic retrieval

A vector or graph index over all the layers, queried per-turn for the current topic. Two rules that saved us:

1. Retrieval is a bonus, never the source of truth. If retrieval contradicts the curated file, the file wins.
2. Fast beats smart. A retrieval call that takes 9-20s per turn will silently degrade the whole agent. Run a fast vector mode by default; reserve deep graph traversal for explicit deep-recall requests.

Layer 5 — Write-back

After every successful turn, new facts flow back down: raw log always, retrieval index asynchronously, occasionally a promotion into short/long-term. Without write-back, memory only decays.

The contract between layers

This is the part that makes it survive months instead of days:

• Layers cite downward. A summary references the raw-log dates it came from, so any claim is traceable to ground truth.
• Layers never trust upward. Retrieval output is checked against curated facts, not the other way around.
• Every layer can die independently. Retrieval service down? The agent still has layers 1-3 in context. Summaries stale? Retrieval still finds the raw log. A blank-slate failure requires all five to fail at once — which is why it effectively never happens.

Minimal implementation (start here)

You do not need a vector database to get value. Build layers 1-3 in an afternoon:

memory/
  2026-07-05.md        # layer 1: append every turn, never edit
  SHORT_TERM.md        # layer 2: rewritten nightly by a cron + your model
  LONG_TERM.md         # layer 3: curated by hand + promotions from layer 2

A daily cron reads yesterday's raw logs and rewrites SHORT_TERM.md:

# nightly, e.g. via crontab
cat memory/$(date -d yesterday +%F).md \
  | your-llm "Rewrite SHORT_TERM.md: a 3-5 day rolling summary of active
     projects, open loops, and decisions. Keep it under 40 lines." \
  > memory/SHORT_TERM.md

Then load SHORT_TERM.md + LONG_TERM.md into your agent's context on every turn. That alone gets you ~80% of the value. Add retrieval (layer 4) only when your logs outgrow the context budget — not before. Most assistants never need it as urgently as they think.

Run this repo

This repo isn't just the diagram — all five layers are here as working, dependency-free Python (memstack/). Standard library only, so it deploys anywhere Python 3.10+ runs.

git clone https://github.com/stas4000/memory-stack && cd memory-stack
python examples/agent_loop.py     # full loop around a stub LLM, no keys needed
python tests/test_memstack.py     # 7 self-checks, one per layer contract

Use it as a CLI (the LLM is any command that reads stdin, writes stdout — so no SDK and no keys live in this repo):

memstack log user "book flights to Berlin next week"   # layer 1: raw log
memstack search "the Berlin trip"                      # layer 4: recall
memstack promote "user prefers window seats"           # layer 3: curate
memstack context                                       # layers 1-4: per-turn block
memstack summarize --cmd "ollama run llama3" --days 3  # layer 2: nightly cron

Or from Python:

from memstack import MemoryStore, recall

store = MemoryStore("memory")
store.log("user", "the Berlin trip is confirmed for August")
context = store.build_context(retrieval=recall(store, "Berlin"))  # feed to your agent
store.log("assistant", reply)                                     # layer 5: write-back

File	Layer it implements
memstack/store.py	1 raw log, 2 short-term, 3 long-term, 5 write-back
memstack/retrieval.py	4 retrieval — keyword (zero-dep) + vector adapter
memstack/summarize.py	2 nightly rewrite + 3 promotion, any LLM
memstack/cli.py	all five, from the shell

Retrieval ships two backends behind one interface: a pure-stdlib TF-IDF cosine that most assistants never outgrow, and a VectorRetriever where you plug your own embed() when you actually need it. No vendor lock, no keys.

Why five and not one

Every single-system memory has a failure mode that looks like the agent "going senile": it forgets a standing preference, or confidently recalls something that never happened, or blanks entirely after a service hiccup. Layering converts those from outages into degraded-but-working states. The agent is never smarter than its best layer, but it's also never dumber than its most reliable one.

---

Part of a set of production agent patterns. See also: operator-agent-instructions, autoresearch-loop, judge-alignment.

MIT licensed — take what works. Built by Bles Software. I post what we learn running these systems: @stas_sorokin_.