Benchmarks
Published decision-recall benchmark
A fixed-corpus experiment comparing configured retrieval paths. The setup and limitations below are part of the result.
Pinned corpus (v3)
Corpus hash: f1bb11060a3a59fb · Run: 20260710T114611Z
These scores compare top-one baselines with a Neither path given source paths from the test fixture. They do not measure an end-to-end MCP conversation or establish superiority over RAG systems generally.
Neither
98%Naive pgvector
44%mem0ai/oss
44%Delta vs naive: +54pp · Delta vs mem0ai/oss: +54pp
By question category
| Category | Neither | Naive | mem0ai/oss |
|---|---|---|---|
| Flat Fact | 100% | 89% | 89% |
| Multi Hop Why | 100% | 11% | 11% |
| Rejected Approach | 100% | 14% | 14% |
| Bc Isolation Trap | 91% | 45% | 45% |
| Supersession | 100% | 22% | 22% |
| Partner Unseen | 100% | 100% | 100% |
Neither decision recall 98% vs naive pgvector 44% and mem0ai/oss 44% on pinned corpus f1bb11060a3a59fb.
Methodology
The pgvector baseline selects one snippet by embedding similarity. The mem0ai/oss baseline also selects one result, with memory inference disabled (infer: false). Both operate without the test fixture’s business-context filter.
The Neither path receives the question’s known seed source and related source paths from the fixture graph. The harness calls file-context retrieval for those paths. In the database-enabled path, it then scores text fetched directly from the ingested sources, even if a file-context call fails. Required decision facts are also supplied as retrieval hints.
A pass means the required facts occur in the selected text. It does not score a generated answer, a fresh-conversation MCP workflow, or retrieval from an unknown source. The systems receive different guidance and context budgets; broader RAG performance requires a separate comparison with matched inputs and retrieval limits.
Published 2026-07-10 on pinned corpus v3 — reproduce with pnpm benchmark:decision-recall:real:live and pnpm verify:decision-recall-benchmark.