Reconstructive Memory for Learning Systems (v0)

Jan 2026

Modern learning systems are getting increasingly better at reasoning, planning and acting however solutions aiming to represent their memory are becoming increasingly patchworked: longer context windows, vector databases, “reflection” summaries, replay buffers, world models, knowledge graphs. Even though all of these components might help, there seems to be some standard failure modes: drift (context rot), brittle recall, confabulation, false overconfident “memories” that don’t survive scrutiny.

I believe it is important for the future of the ML field that the memory problem is not implicitly solved, but clearly defined, formalized, tested and evaluated.

The magic thing about ML is that many times you can just look back at how humans work for the solution.

The following is a Cognitive Science - based approach for framing the memory problem for learning systems.

“Treat memory in learning systems as reconstructive inference”

Reconstructive Memory in Cognitive Science

A practical (and proven) way to understand human memory is as lossy compressions plus reconstruction. We store a compressed trace of experience (not every pixel and token) and later we reconstruct what we need for the current situation.

Reconstruction uses three ingredients:

• Cue: whatever the system uses as a handle to decide what to retrieve and how to interpret it (a question, a goal, the current situation)
• Trace: whatever was stored
• Schema: the learned structure that tells us what’s typical, causal, or relevant

From this perspective, distortions aren’t mysterious. If the trace is missing detail, reconstruction will lean more on the schema. If the cue is leading or adversarial, it can steer the reconstruction. If provenance (i.e. “where did this come from?”) is unclear, the reconstructed output can feel confident without being verifiable.

Formalization

With that lens, we can name the minimal pieces involved in recall and make the reconstruction problem explicit.

Components

• Episodic trace z : what was stored from past experience (often compressed).
  Represents long-term memory
  Examples:
  • LLM agents: retrieved documents, tool logs, chat history chunks
  • Perception / robotics: sensor snapshots, map fragments
  • VLMs / VLAs: short clips, keyframes, latent summaries of observations
  • RL: replay buffer samples, trajectories


• Schema / prior s : the learned structure that interprets traces and fills gaps.
  Represents “intuition”: what the system believes is typical, causal or likely.
  Examples:
  • Model parameters / weights
  • Learned world model or transition model
  • Learned task abstractions and skills
  • Knowledge graphs
  • Distilled summaries


• Cue c : the current question/goal + context that triggers recall and shapes what gets reconstructed.
  Represents working memory: it decides what’s relevant right now
  Examples:
  • User prompt / query
  • Current robot state and observation
  • Planner subgoal
  • A detected anomaly that triggers an investigation


• Reconstruction x : the “memory content” the system produces for reasoning and action.
  The usable belief derived from the trace.

Under this framing, recall is not “read memory -> output”. It’s closer to inference:

p(x | z, s, c)

Meaning: the system forms a belief x by combining:

• evidence from stored traces z,
• learned structure s,
• and the current cue c that determines what to retrieve and how to interpret it

This small equation forces a useful question:

When a system “remembers”, how much of that output is trace-grounded, and how much is the schema filling in the blanks?

Epilogue

This post is meant as a gentle starting point, not a closed theory. The goal is to make one thing explicit: memory in learning systems is already reconstructive in practise, and we should treat it as a first-class research problem rather than an implicit byproduct of scaling context windows or bolting on retrieval.

In a follow-up, I want to turn this framing into something operational:

• a concrete taxonomy of failure modes (schema-dominant reconstruction, trace fragmentation, provenance collapse)
• evaluation protocols that score not only correctness, but grounding, uncertainty, and provenance.

If this framing resonates and you’re working on long-horizon agents, robotics, or continual learning, I’d love to compare notes; especially on where current systems fail in the wild, and what would count as a benchmark for “memory that stays useful over time”.