Long context is not memory. We measured Mem0 on our fixtures.

A memory layer has to add a fact once, find it later, and do both fast enough that the agent does not notice. We re-ran our keyless Mem0 scorecard on the persona fixture corpus on 2026-10-13 (mem0ai==2.2.1, MEM0_TELEMETRY=False) and got add p50 6.5 ms, search p50 3.4 ms, and recall@5 of 1.0 across 9 memories and 9 queries. The catch is that these are our numbers, from our machine, on nine hand-written fixtures, so read them as a wiring check and a baseline rather than a parity claim.

The catch

The scorecard is a fixture-corpus measurement. It makes no parity claim against Meta's Muse Wonder, Mem0's managed platform, or the scores in the upstream README. Nine memories and nine queries is a smoke test. Recall@5 of 1.0 on nine hand-written persona facts tells you the embedding and index are wired correctly; it tells you nothing about how the layer behaves on a thousand noisy chat turns.

The latency is also narrow by design. The keyless path calls Memory.add(..., infer=False), so no LLM ever extracts or rewrites facts. The numbers are FastEmbed (all-MiniLM-L6-v2) plus a local FAISS index on a laptop. The optional LLM bench (include_llm_bench: false) stays off and the report records llm_bench: null. Turn it on with an OPENAI_API_KEY or GEMINI_API_KEY and you are measuring a different system with different costs.

The first run after a fresh uv sync was cold (add p50 around 28 ms while FastEmbed pulled its model into cache). The second run, which is the one in the report, is warm. Expect the same shape on your machine, and compare warm to warm.

What we ran (keyless)

uv sync
MEM0_TELEMETRY=False uv run memory-bakeoff bench run --json '{"top_k":5,"include_llm_bench":false}'

The run needs no API keys, makes no Ollama round-trips (the Ollama provider is configured but never called when infer=False), and sends no telemetry. From reports/scorecard.json (run on October 13, 2026):

Search is cheaper than add at every percentile, which is what you want from a memory layer, since writes happen once and reads happen on every turn. The p95 tails (19.6 ms add, 9.6 ms search) are the numbers to watch if you put this inside an agent loop; a p50 that looks free can still cost you when it spikes.

The same lab also checks the things a scorecard cannot. The mem0ai pin is locked at 2.2.1 (see docs/MEM0_PIN.md), MEM0_TELEMETRY=False is the default in .env.example and CI, and the Bun MCP server in mcp/ typechecks and exposes remember, recall, forget, and scorecard_path over stdio. uv run pytest tests -q passes keyless.

How it is wired

Mem0 sits between the app and the vector store. The app calls add and search; Mem0 embeds, indexes into FAISS, and optionally asks an LLM to infer or merge facts. The keyless path removes the LLM from the loop, so the numbers describe the memory layer itself. That is the slice we measured.

If you are choosing a memory product, run the same fixtures on your hardware and read your own p95. Latency and recall quoted without a shared corpus tell you nothing about your workload. The scorecard here is small on purpose. Extend it with your own jsonl fixtures and quote those results.

The tool

mem0ai/mem0

mem0ai/mem0 is a memory layer for agents: 67,109 GitHub stars, Apache-2.0, Python (fetched 2026-10-13).

Minilab (open source)

Clone it, run the one-liner above, and open reports/scorecard.md. Swap the fixture file under src/memory_bakeoff/fixtures/ for your own facts and queries, re-run, and you have a bakeoff on your data in under a minute.

uv sync && MEM0_TELEMETRY=False uv run memory-bakeoff bench run --json '{"top_k":5,"include_llm_bench":false}'