agent-readings

Memory and State

MemGPT

MemGPT treats the context window as RAM and external stores as disk, then gives the model functions for paging memory in and out.

7Claims
7Evidence

Why It Matters

The paper turns memory from a retrieval add-on into runtime state. For agent infrastructure, the key move is that memory operations become logged, inspectable actions with their own policy and failure modes.

Problem

  • Fixed context windows limit long conversations and document analysis.
  • Blindly extending context is expensive because attention cost grows quadratically and long-context models may still fail to use middle context reliably.
  • Naive summarization loses details that a later agent step may need.
  • Retrieval-augmented generation typically retrieves into context but does not let the model manage a persistent memory hierarchy over time.

Key gap: The paper asks how to make a fixed-context model behave as if it had larger, persistent memory without retraining the model or increasing the context window.

Method

Wrap a fixed-context LLM in an OS-like runtime that stores prompt-resident state in main context, out-of-context state in recall and archival storage, and exposes memory-management functions the model can call.

  • Store static instructions, writable working memory, and recent events in main context.
  • Append user and model messages to recall storage so evicted history remains queryable.
  • Store long documents or important facts in archival storage, implemented in experiments with PostgreSQL, pgvector, and HNSW indexing.
  • Insert memory-pressure warnings before context overflow so the model can preserve useful information.
  • Flush old FIFO messages when needed, keeping a recursive summary plus durable recall storage.
  • Parse model outputs as memory functions, execute them, feed results/errors back, and optionally continue via function chaining.
  • Evaluate the same runtime in multi-session chat, document QA, and nested key-value retrieval.

Claims

C1 · paper-supported

MemGPT frames the context window as main memory and external stores as disk-like memory, with the model responsible for moving information between tiers through function calls.

Scope: Introduction and Section 2.

  • E-memory-hierarchy: The paper proposes virtual context management: main context holds prompt-resident state, external context holds out-of-context data, and the LLM uses functions to retrieve, write, and evict memory across tiers.

C2 · paper-supported

The runtime separates system instructions, writable working context, FIFO queue, recall storage, and archival storage, and uses memory-pressure warnings plus flushing to control overflow.

Scope: Sections 2.1 and 2.2.

  • E-main-external-context: Main context is split into system instructions, writable working context, and a FIFO queue. The queue manager writes messages to recall storage, warns near context overflow, flushes messages at the flush threshold, preserves a recursive summary, and keeps evicted messages in recall storage.

C3 · paper-supported

MemGPT makes memory management event-driven and supports function chaining so the model can run multi-step memory searches before yielding control.

Scope: Sections 2.3 and 2.4.

  • E-control-flow: The function executor parses model outputs as memory function calls, feeds results and runtime errors back into context, and supports immediate follow-up inference through a heartbeat flag or yielding until the next event.

C4 · paper-supported

On deep memory retrieval over Multi-Session Chat, MemGPT substantially improves accuracy and ROUGE-L over fixed-context baselines for GPT-3.5, GPT-4, and GPT-4 Turbo.

Scope: Section 3.1.1, Table 2.

  • E-dmr: DMR accuracy and ROUGE-L improve from GPT-3.5 Turbo 38.7% / 0.394 to MemGPT 66.9% / 0.629, GPT-4 32.1% / 0.296 to MemGPT 92.5% / 0.814, and GPT-4 Turbo 35.3% / 0.359 to MemGPT 93.4% / 0.827.

C5 · paper-supported

On document QA, MemGPT uses archival vector search to page through more documents than fit in prompt context, but performance remains tied to retrieval quality and model function-calling ability.

Scope: Section 3.2.1 and Figure 5.

  • E-document-qa: The document-QA setup loads a late-2018 Wikipedia dump into archival storage, uses OpenAI text-embedding-ada-002 embeddings with PostgreSQL, pgvector, and HNSW, samples 50 NaturalQuestions-Open questions, and shows MemGPT paging through retrieval results while fixed-context baselines are capped by the documents placed in prompt context.

C6 · paper-supported

On nested key-value retrieval, fixed-context baselines fail as nesting grows, while MemGPT with GPT-4 remains effective by chaining repeated memory queries.

Scope: Section 3.2.2 and Figure 7.

  • E-nested-kv: Nested KV uses 140 UUID key-value pairs, varies nesting from 0 to 4, and samples 30 orderings. GPT-3.5 reaches 0% accuracy at 1 nesting level; GPT-4 and GPT-4 Turbo reach 0% by 3 nesting levels; MemGPT with GPT-4 remains unaffected by nesting depth in the paper's description.

C7 · paper-supported

The approach is bottlenecked by the base model's ability to call functions, manage search depth, and continue paging when the needed evidence has not yet been retrieved.

Scope: Sections 3.2.1 and 3.2.2.

  • E-model-dependence: The authors report degraded document-QA performance with GPT-3.5 because of limited function-calling ability, note that MemGPT may stop paging before exhausting retrieval results, and observe that MemGPT with GPT-3.5 and GPT-4 Turbo drops off in nested KV because the model fails to perform enough lookups.

Agent Infrastructure Takeaways

  • A context window is main memory, not durable memory. Persistent agent state needs explicit storage tiers and movement rules.
  • Memory actions should be first-class runtime events: search, write, flush, summarize, page, and heartbeat all need logs and policy.
  • Memory pressure is a useful control signal. Long-running agents should know before context eviction happens, not after facts disappear.
  • The memory manager is only as reliable as the base model's function-calling and stopping behavior.
  • RAG is not enough for long-horizon agents. The model also needs control over when to retrieve, how many pages to inspect, and when to save new state.
  • External memory creates infrastructure obligations: database schema, vector indexes, pagination, provenance, compaction, and access control.

Caveats

  • The paper's memory policy is prompt- and model-driven rather than formally specified or verified.
  • Several evaluations rely on LLM judges or similarity metrics, not executable ground truth.
  • The document-QA experiment uses a small sample of 50 questions.
  • Performance depends strongly on model function-calling quality; GPT-3.5 is notably weaker in the paper's experiments.
  • MemGPT can stop searching before reaching the needed evidence, so the illusion of infinite context is bounded by search policy.
  • The paper does not deeply address privacy, access control, retention policy, or deletion semantics for long-term user memory.