The question that matters: does answering from Coalent's cached understanding hold answer quality against the retriever it sits on top of — especially the numbers — while spending far fewer context tokens, attributing every figure to the right source, and answering questions a single retrieval pass can't?
So the benchmark refuses every shortcut that would flatter us: no LLM-judge (which self-prefers), a retriever shared by both arms so we can't win on retrieval, and grading with escalation off so we measure the cache alone. Two runs live on this page: a real-world run on MultiHopRAG with third-party gold questions, and our own structured-regime harness (synthetic templates).
Real-world benchmark (news corpus, n=605)
New in v0.6 This run isn't our own harness: 609 real news articles with third-party gold questions (the MultiHopRAG corpus). Answerer gpt-4.1-mini, strict grading (normalized gold containment in the answer), the identical embedder in both arms, 605 frozen held-out questions, and naive's own token-scaling curve measured on the same stream.
Headline: the v0.6 pool read path matches chunk-RAG's best measured accuracy at 25–43% fewer context tokens (n=605, strict grading).
The fairness control we run that most benchmarks don't: naive's own token-scaling curve, on the same stream — so Coalent is compared against every operating point of the baseline, not one convenient k:
| Arm | Accuracy | Context tokens / read |
|---|---|---|
| naive k=4 | 0.582 | 590 |
| naive k=6 | 0.638 | 882 |
| naive k=9 | 0.711 | 1,311 |
| naive k=12 (best measured) | 0.731 | 1,729 |
Coalent pool (read_path="pool", v0.6, warmed cache) | 0.731 | 981 |
- Fewer tokens at naive-k9 accuracy: 0.731 @ 981 vs k9's 0.711 @ 1,311 — ~25% fewer context tokens (CIs overlap: parity, not improvement).
- Naive's best measured point at 57% of the budget: 0.731 @ 981 vs k12's 0.731 @ 1,729 — ~43% fewer tokens at equal accuracy. This is the strongest efficiency framing, and it is the same parity logic.
- Serving ranks: the gold claim's rank in final pool order lands at p50/p75/p90 = 1/6/15 (over claim-present queries,
reranker=None— the plain cosine default met the targets set for the reranker tier). - The behavioral stack (default-OFF: residual spans + refusal fallback + query keys, driven through the full loop): final 0.769 @ 1,003 tokens vs the same-store default's 0.737 @ 982 — +3.1 pts for +2.1% tokens, refusals 91 → 61 (−33%), zero newly-wrong answers. Requires wiring
report_refusal/report_successinto your answer loop. - Attribution headers are a measured ladder: opaque id 0.641 → shipping query-title default 0.678 → caller-metadata callable 0.731. The best rung needs your corpus metadata — the pool_header recipe is the documented golden path; the residual gap is a unit-metadata limit, not header format.
- The point held across versions: the v0.5 preview's 0.699 @ ~1,036 anchor and v0.6's 0.731 @ 981 have overlapping CIs on the same rig.
- Economics (from the v0.5 build-layer pilot on this corpus): extractor build spend ~$0.14–0.20 per ~600-read stream (gpt-4o-mini); break-even ≈ 4–5 reads per source; widened units read a median of 23 chunks of their source vs 2 for keyhole builds; rebuild churn fell 460 → 31.
The honest framing. We do not claim to beat naive RAG on accuracy — a pre-registered decisive at the 430-token point found no beat and no dominance region on the measured curve, and naive's own curve reaches the same accuracy given more tokens, so we do not claim compression either. The claims are: equal accuracy at 25–43% fewer context tokens, plus what naive cannot do at any k — freshness (stale claims invisible the instant mark_dirty runs), provenance, and behavioral compounding on repeat traffic. Known limits, measured: the refusal fallback flips ~20% of natural refusals (33% when the payload contains the answer verbatim — the 68% lab figure holds only for span-derived questions); query keys can fire on sibling articles in dense same-topic corpora (3/605 observed, answers still correct — raise key_floor there); the shipping default header refuses per-outlet questions its payload cannot attribute (wire pool_header to recover ~5 pts). Everything here is measured on news/document corpora with factual queries — we make no claim about conversational or agent-memory workloads.
Structured-regime benchmark (synthetic templates)
Our own harness, from the v0.4/v0.5 build-layer era — the regime Coalent is built for: facts that get re-asked.
Setup
- Structured/reuse workload — 64 sources × 3 seeds = 192 reads per condition.
- Real OpenAI embeddings — no offline hashing stand-in that could inflate routing.
- A deterministic accuracy check — a literal number + attribute match against gold. No gpt-4o judge grading its own family's answers.
- A real dense top-5 retriever, shared by both arms. Naive RAG is that retriever, re-read every query. Coalent sits above the same retriever. Whatever Coalent gains is not a retrieval advantage.
- Graded escalation-off. Coalent's RAG floor (append raw retrieval when coverage is low) is disabled during grading, so the number reflects the cache's own understanding — not a fall-back to full context.
- Twin-free corpus with asserted invariants — every identity resolves to one value (see the transparency note below for why this matters).
Results
On the synthetic template corpus, answer accuracy is at parity across four answer models — the 95% confidence intervals overlap in every row — while Coalent sends a fraction of the context.
| Answer model | Naive dense RAG | Coalent |
|---|---|---|
| gpt-4o-mini | 0.81 | 0.81 |
| gpt-4.1-mini | 0.90 | 0.85 |
| gpt-4o | 0.90 | 0.87 |
| gpt-4.1 | 0.99 | 0.97 |
| Context tokens / read | 126 | 47 |
Same retriever, same gold, same deterministic check — Coalent lands on top of naive RAG's accuracy while carrying roughly a third of the context per read. The extractive understanding (a query-independent list of source-grounded claims) keeps the figures that a prose summary would drop.
Trustworthy, not just cheap
Cheap is easy if you're willing to answer from the wrong source. Coalent isn't.
- route@1 ≈ 1.00 — the cache picks the correct unit for the query essentially every time.
- Misattribution ≈ 0–2% — serving a number from the wrong source is down at the level of the naive answerer's own noise. When Coalent gives you a figure, it came from the source it says it did.
- Multi-hop: naive 0% → Coalent 100% (a template-fixture result — synthetic bridge questions; see the real-world run above). Questions whose answer requires bridging two facts from two different sources are structurally impossible for a single top-K retrieval pass. Coalent's cross-unit claim recall (the unit path's multi-hop mechanism, on by default) sweeps per-claim memory across all fresh units with MaxSim and surfaces the bridge fact — at zero extra LLM calls.
Economics
Building a unit costs about 430 tokens and ~4s per source, once. After that every read is a cheap embedding lookup instead of a full-context re-read.
- Break-even at ~4–5 reads per source. On a reuse corpus (192 reads over 64 sources) you clear that on almost every source.
- Cheaper on every read forever after — 47 vs 126 context tokens, no re-synthesis, freshness handled by dirtying the single affected unit instead of rebuilding a graph.
This is the opposite of GraphRAG's build-the-whole-graph-upfront tax: units are built lazily, only when a query needs one.
The honest read. On the structured-regime (synthetic template) benchmark, answer quality is on par with naive dense RAG — the CIs overlap on all four models — at roughly a third of the context tokens per read, with the right-source guarantee (route@1 ≈ 1.00) that raw retrieval doesn't give you. The multi-hop capability gap — naive 0%, Coalent 100%, at zero extra LLM calls — is a template-fixture result; the real-world MultiHopRAG run above is the uncontrived version of the same story. Parity accuracy is the floor here, not the headline.
Transparency — the misattribution scare that was a benchmark bug. An earlier internal run reported a large misattribution rate, and we chased it as a routing failure. It wasn't. The corpus contained contradictory duplicate sources — two "facts" asserting different values for the same identity. No router can resolve that; it's a coin flip, and a zero-residual coin-flip ceiling fit confirmed the error rate was exactly the pigeonhole artifact, not a model weakness. We fixed the corpus (identity-unique, with asserted invariants), not the metric — and route@1 went to 1.00 with misattribution to ~0–2%. We're telling you this because a benchmark you can't audit isn't worth quoting. Our earlier token-only, independent-judge benchmark is superseded by the numbers above and no longer represents Coalent.
Also shipped: a deterministic mechanism check
For a no-API-key, deterministic check of the freshness/cost mechanism (no LLM), the package ships run_benchmark:
from coalent.evaluation import run_benchmark
for name, report in run_benchmark().items():
print(name, report.accuracy, report.stale_rate, report.cost_tokens)
Next
- How it works — the mechanism behind the numbers.
- Provenance & freshness — why the stale-rate is zero.