The metric problem
A symmetric score rewards a memory that answers everything — including the questions its store cannot support. VeriBench scores memory the way a deployment pays for it: NET(λ) = (correct − λ·wrong) / n, where λ is the declared cost of a wrong answer relative to a silence. Legal and medical run at high λ; brainstorming runs low. One number, swept over λ ∈ {1, 2, 5, 10} — fixed before any run.
✓ No LLM judge, no network, no leaderboard theater — one command, raw JSON committed to the repo.
Memory benchmarks measure how much comes back. None of them price what comes back wrong.
On the answerable half of a corpus, a gated memory and a raw vector store retrieve almost identically — recall@k sees no difference. The difference appears on the questions the store cannot answer: a memory without an abstention floor returns its nearest neighbour anyway, with full confidence. In production that is a fabricated answer; in a symmetric benchmark it is invisible.
NET(λ) makes it visible by construction: a correct answer earns +1, a wrong one costs λ, silence costs nothing. The break-even accuracy is λ/(1+λ) — the same threshold Verimem’s SLA knob tunes the store at. The benchmark measures the store at the exact number the operator tunes it with.
Hypothesis, metric, λ sweep and refutation conditions are committed in PREREGISTRATION.md before any run. The scoring and the outcome mapping were written and unit-tested first — a favourable result cannot be manufactured by choosing the metric after the fact.
Real datasets we did not write: HaluEval QA and SQuAD v2 (answerable + unanswerable splits, disjoint). Correctness is id-decidable retrieval — no LLM judge anywhere in the loop.
A scrambled store must go deeply negative (it does: NET(1) = −0.94) and the identical retrieval with the floor off must fabricate on unanswerables (it does: 100 wrong answers). If a control passes, the benchmark is broken — that is the point of having it.
Head-to-head runs use the identical embedder (multilingual-e5-base) on both engines, offline, mem0 in raw-store mode (its LLM is never called — that axis is out of scope by declaration, not omission).
| System | ✓ | ✗ | ∅ | coverage | NET(1) | NET(2) | NET(5) | NET(10) | goes negative at λ |
|---|---|---|---|---|---|---|---|---|---|
| Verimem · floor τ=0.8 (product default) | 182 | 4 | 114 | 0.62 | +0.593 | +0.580 | +0.540 | +0.473 | 45.5 |
| mem0 2.0.11 · as shipped (no floor) | 200 | 100 | 0 | 1.00 | +0.333 | 0.000 | −1.000 | −2.667 | 2.0 |
| mem0 + bolted floor 0.75 (tuned on the eval) | 166 | 0 | 134 | 0.55 | +0.553 | +0.553 | +0.553 | +0.553 | never |
| Same store, floor OFF (τ=0 control) | 192 | 100 | 8 | 0.97 | +0.307 | −0.027 | −1.027 | −2.693 | 1.9 |
| Scrambled control (sanity: must fail) | 5 | 287 | 8 | 0.97 | −0.940 | −1.897 | −4.767 | −9.550 | 0.02 |
Read it honestly, both ways: as shipped, mem0 turns net-negative past λ=2 — 100 fabricated answers on the unanswerable half; Verimem’s product default stays positive out to λ≈45. But a floor can be bolted onto any engine: with a threshold tuned on this eval, mem0 reaches a flat +0.553 — which beats our default at λ≥5 on this corpus. The differences that remain: the engine itself ships no floor, the bolted threshold was chosen on the test set, and the flat line means it answers nothing it isn’t sure of — coverage 0.55 vs our 0.62 with 182 vs 166 correct at λ=1.
| System | ✓ | ✗ | ∅ | coverage | NET(1) | NET(2) | NET(5) | NET(10) | goes negative at λ |
|---|---|---|---|---|---|---|---|---|---|
| Verimem · floor τ=0.8 (product default) | 163 | 49 | 88 | 0.71 | +0.380 | +0.217 | −0.273 | −1.090 | 3.3 |
| Verimem · best floor 0.85 (tuned on the eval) | 98 | 7 | 195 | 0.35 | +0.303 | +0.280 | +0.210 | +0.093 | 14.0 |
| mem0 2.0.11 · as shipped (no floor) | 200 | 100 | 0 | 1.00 | +0.333 | 0.000 | −1.000 | −2.667 | 2.0 |
| mem0 + bolted floor 0.80 (tuned on the eval) | 44 | 0 | 256 | 0.15 | +0.147 | +0.147 | +0.147 | +0.147 | never |
SQuAD’s distractor passages compress the score band, and it shows: at the product default the crossover drops to λ≈3.3, and holding NET positive at λ=10 costs coverage 0.35. Abstention is a dial, not magic — the corpus decides how expensive honesty is. The wrong move would be hiding this table.
The tables above: external data, answerable + unanswerable, floor on vs off vs competitor, scrambled control. The axis where fabrication becomes a priced event.
A store that faithfully corroborated a spurious correlation answers do(X) questions confidently wrong — and nets negative even at λ=1. Provenance is not causality; a benchmark that can’t distinguish them will reward the confusion.
Collusion (N identities, one feed) plus a trusted sleeper. Only a two-channel trust policy — independent corroboration and outcome feedback — stays net-positive; each single channel fails exactly one of the two attacks. The axis behind Verimem’s two-channel source trust.
every axis, deterministic, no API keys
git clone https://github.com/aureliocpr-ctrl/verimem && cd verimem
pip install -e .
python -m benchmark.veribench.run_all # every axis, one reproducible entrypointSpec, pre-registration and every raw result JSON live in the repo under benchmark/veribench/ and benchmark/results/. Competitors are invited — the mem0 adapter is in-tree as the worked example; PR your engine’s official adapter and we run it on the same footing.