Every get(query) walks the same fixed ladder of gates, top to bottom: match a unit, keep it fresh, cover the query, serve the minimum. Each rung is also a knob — but you rarely touch one. The defaults are pure cosine: the whole ladder runs with no extra model and no extra dependency, and only reaches for raw when a hit genuinely under-covers.
The default path costs nothing extra. Gates 1–8 are cosine over embeddings you already computed — no cross-encoder, no NLI, no second LLM call on the read path. You turn a rung up only when a specific workload needs it (see the table below), and one line reverts to exact v0.3 behaviour.
New in v0.4 Two rungs changed in v0.4: the matched unit is now a list of query-independent extractive claims (see The Synthesizer), and cross-unit recall is on by default (gate 5) — so multi-hop questions resolve for free on the read path.
The ladder, in firing order
A read flows down the spine — matched, fresh, covered, served. Each gate can branch off: a miss builds a new unit, a stale unit re-materializes, recall pools claims across units for free, and a genuinely thin hit escalates to the raw evidence (the RAG floor). It never fabricates.
The same eight gates as a precise reference:
| # | Gate | Default | Fires when → what happens |
|---|---|---|---|
| 1 | hit_threshold | auto (~0.33 OpenAI) | best unit's blended score 0.7·topic + 0.3·seed is below the floor → MISS: retrieve + synthesize a new unit |
| 2 | hit_margin | 0.0 (off) | top unit beats the runner-up by less than the margin → ambiguous: build the query's own unit instead of guessing |
| 3 | freshness | always on | matched unit is dirty or expired → re-materialize it before serving |
| 4 | coverage | max per-claim cosine | always computed — how well the matched unit covers the query; gates 5–7 read this value |
| 5 | cross_unit_recall | on | coverage < recall_threshold → pool the best claims across all fresh units (MaxSim) and surface bridge facts; lifts coverage, no LLM call |
| 6 | coverage_scorer (S2) | None (off) | coverage lands in [coverage_floor, coverage_ceiling) → a cross-encoder / NLI / one-token-LLM check overrides cosine |
| 7 | coverage_floor | auto (~0.28 OpenAI) | coverage still below the floor → ESCALATE: append fresh raw retrieval (no LLM call) — the RAG floor |
| 8 | select_floor | None (lexical trim) | serve the atoms whose per-claim cosine ≥ floor — the query-relevant claims by meaning, fewer tokens |
A hit that survives gate 1–2, is fresh (3), covers the query (4–7), and is trimmed (8) is served straight from the cache. A miss at gate 1 or an ambiguous tie at gate 2 builds a new unit; an under-covered hit at gate 7 escalates to fresh raw — never a fabricated answer.
Before the ladder: residual_floor (build time)
residual_floor runs once, at build, not on the read path. When the extractor drops a number-bearing source span — its max per-claim cosine falls below the floor — that span is kept verbatim as an extra atom bound to the unit, closing the extractor-recall gap for stray numbers. residual_limit (default 24) caps how many are kept, ranked most-missed first, so the safety net can't defeat the token win. Embedding-only, no extra LLM call; None = off.
Other hooks on the read path
These sit alongside the ladder rather than on it:
route_by_claim(default off) — route the match on the best per-claim embedding instead of the unit's topic embedding; useful when one unit holds many distinct facts.relevance_gate— a(query, chunks) -> chunkscallable that drops irrelevant retrieved chunks before synthesis, so a new unit is built from cleaner evidence.depth— on theLLMSynthesizer(0.0terse →0.5balanced default →1.0exhaustive); trades synthesis cost against how often later reads escalate. See The Synthesizer.calibrate_thresholds(embedder, positives, negatives)— derivehit_threshold/coverage_floorfrom labeled(query, understanding)pairs for a custom ortext-embedding-3-largeembedder; the shipped embedders are auto-calibrated already.
Which knob for which workload
| Workload | Reach for |
|---|---|
| Multi-hop — facts split across sources | recall_threshold ≈ 0.7 (cross-unit recall is already on) |
| Contradiction / collision-heavy corpora | hit_margin > 0 — costs rebuilds; leave off on clean data |
| Paraphrase-heavy queries over large units | select_floor — serve claims by meaning, fewer tokens |
| Messy real-world prose, numbers buried in text | residual_floor — recover extractor-missed number spans |
| High-stakes ambiguity | coverage_scorer (S2) — containment-grade check in the borderline band |
| Reproduce exact v0.3 behaviour | extract=False + cross_unit_recall=False |
hit_margin and coverage_scorer cost you something — extra rebuilds and an extra check respectively — so they earn their place only on ambiguous or collision-heavy data. On clean, structured corpora the pure-cosine defaults are already at parity; don't turn rungs up you don't need.
Reading the ladder in production
s = cache.stats()
s["hit_rate"] # share of reads served from cache
s["escalation_rate"] # share that hit the RAG floor (gate 7)
s["hit_threshold"], s["hit_margin"], s["coverage_floor"]
s["recall_threshold"], s["select_floor"], s["residual_floor"]
cache.stats() reports hit_rate and escalation_rate alongside the effective operating thresholds — so you can see which rung is doing the work and confirm exactly which knobs are live. A climbing escalation_rate means understanding is too thin (raise depth); a low hit_rate on paraphrases of the same question means the match bar is too high (hit_threshold).
Next
- The Synthesizer — where extractive understanding and
depthcome from. - get() & Result — the single read API and every field it returns.