Memory Bake-off: test Mem0, Zep and plain search on your own agent logs before you buy

Several open-source harnesses already compare memory backends side by side: Supermemory's MemoryBench, memeval, memtrust and EverMind's framework. Almost all of them are run by memory vendors, and they score public benchmarks like LoCoMo and LongMemEval rather than the customer's own conversations. Those benchmarks have contested methodology, and the LoCoMo license restricts commercial use. None of the sources I found offered a neutral product that does the full workflow end to end: import production chat or agent logs, generate recall questions from them (including contradiction and time-changing facts), then report accuracy, tokens, latency and cost against a plain full-history or grep/agent-search baseline. Several sources explicitly say "run your own eval on your own data" and note that independent cross-vendor testing is thin, which is evidence of real demand. A new entrant could win on vendor neutrality, on generating questions from private logs with privacy safeguards, and on cost-per-correct-answer reporting that includes the plain-search baseline. The risks are that MemoryBench can add a bring-your-own-logs mode cheaply, that general LLM-eval platforms like Braintrust or Langfuse can approximate this with custom datasets, and that buyers may treat it as a one-time evaluation rather than a recurring subscription. Continuous regression testing after a memory system is adopted may be the more durable thing to sell.

• Supermemory MemoryBench

  Open-source harness from the memory vendor Supermemory. It has provider adapters for Supermemory, Mem0 and Zep, runs LoCoMo, LongMemEval and ConvoMem, and accepts contributed providers and benchmarks. It is the closest existing match, but a competing vendor runs it and it centers on public benchmarks rather than your own logs.

• memeval (PyPI memoryeval)

  Open-source pytest-style evaluation framework with adapters for Mem0, Zep, Letta and an in-memory store. It supports custom YAML scenarios, such as checking whether corrections propagate. Scenarios are hand-written; it does not generate questions automatically from imported logs.

• memtrust

  Open-source benchmark harness that runs the same LongMemEval, LoCoMo and contradiction evals against mempalace, mem0, zep and openviking, and publishes the raw logs. Aimed at public benchmarks.

• EverMind unified evaluation framework

  Another memory vendor's framework that compares EverOS, Mem0, MemOS, Zep and MemU through their online APIs, using the same datasets, metrics and answer model. Vendor-run and based on benchmarks.

• Mem0 memory evaluation docs

  Mem0 publishes evaluation guidance and tooling that track accuracy, token cost and latency, but this mostly serves its own benchmark claims.

• Zep evaluation guidance and benchmark critique

  Zep disputes Mem0's LoCoMo methodology and notes that a full-context baseline beat Mem0. This shows that vendor benchmark numbers are contested, which is the core pain point.

• Cognee guide to evaluating AI memory tools

  Compares what each memory tool exposes for evaluation. Cognee is also a memory vendor with its own eval tooling.

Checked 2026-10-22