# Coalent — full reference for AI tools (v0.6.1)

Coalent is a real-time, provenance-invalidated **cognitive cache** for AI agents and RAG.
`pip install coalent` (zero-dependency core). It caches the *understanding* an LLM extracts
from sources — keyed by query **meaning** — retains raw evidence with every unit (so a hit
that under-covers a query falls back to retrieval), and invalidates **surgically by
provenance** the moment a source changes. It sits **above** retrieval (bring any retriever);
it is not itself a retriever. Design axis: the **opposite of GraphRAG** — lightweight,
independent units built **lazily only when a query needs one**, refreshed by dirtying a
single unit (no whole-corpus graph tax).

This file is the complete, accurate reference for generating Coalent code. For prose docs see
https://coalent.ai/docs.

---

## What changed in v0.6.1 — the integrations release (additive; both read paths untouched)

- **The MCP server** — `pip install "coalent[mcp]"` → the `coalent-mcp` console script
  (mcp SDK >= 2.0). Two modes, one seven-tool surface:
  - **PRIMARY: `--cache-factory module:function`** — your factory returns a fully
    user-constructed `SemanticCache` (your vector DB / embedder / LLM / every knob,
    including `store=` and `pool_header=`); the server adds only the protocol. Measured
    byte-identical to the library: 0.7100 strict = the library's own 0.7100 on the same
    100 benchmark questions (609-article corpus; 100/100 served, 0 errors, 98/100 payloads
    byte-equal to the library store). Broken factories fail loud at startup. No
    `OPENAI_API_KEY` demanded — your factory brings its own models. Freshness is
    signal-driven via the `source_changed` tool.
  - **Zero-config: `--watch DIR`** — the recommended v0.6 deployment (pool path, residual
    spans, query keys) over a folder; paragraph-chunked built-in retriever;
    `artifact_id` = relative path; `pool_header` auto-wired to `[{path} | modified
    {YYYY-MM-DD}]` (the golden path by default); store at `<watch-dir>/.coalent/store.db`;
    fully-warm restarts on an untouched folder; requires `OPENAI_API_KEY`, fails loud
    without it. HONEST measured cost: 0.46 vs factory mode's 0.71 on the same questions
    (generic paragraph chunking + keyhole lazy builds) — prefer the factory once you have
    a real index.
  - **Tools (7)**: `get_context(query, budget?)` (folder mode rescans mtime+content-hash
    BEFORE serving — you cannot get a stale answer after saving a file; returns context,
    read_id, sources[], cache_hit) · `report_refusal(read_id)` · `report_success(read_id)`
    · `source_changed(artifact_id, text?)` (BYO freshness signal; hash-skips unchanged
    text; in folder mode the scan stays authoritative) · `list_sources()` ·
    `cache_stats()` · `refresh()`.
  - **Transports**: stdio default (client-launched; SINGLE-WRITER — never point two stdio
    servers at one store) or `--transport http` (streamable HTTP: one long-lived process,
    many agents, ONE shared compounding cache; a single lock serializes tool bodies;
    validated with 2 concurrent clients × 20 interleaved reads identical to sequential,
    zero duplicate synthesis; optional `COALENT_MCP_TOKEN` → bearer auth).
  - Freshness probed at the ANSWER level: mid-run file edits flipped served answers on the
    very next read; zero stale serves observed across the entire validation. Benign
    gate-regime note: a question about a JUST-ADDED file can honestly refuse from a warm
    pool until a build fires for it — nothing stale is ever served.
  - Claude Code: `claude mcp add coalent -- coalent-mcp --cache-factory my_cache:build`
    (or `--watch ./docs`).
- **`langchain-coalent` 0.1.0** — its own package (`pip install langchain-coalent`; deps
  `coalent>=0.6` + `langchain-core>=0.3` ONLY). BYO-first: `create_coalent_cache(
  my_vectorstore, llm=my_chat_model, embeddings=my_embeddings, **knobs)` — every
  `SemanticCache` knob passes through unchanged (new library knobs work the day they
  ship); pool path auto-selected only when a semantic embedder was supplied;
  `CoalentRetriever(cache=)` = drop-in `BaseRetriever` (metadata: read_id / sources /
  cache_hit; `include_evidence=True` adds raw chunks); `CoalentVectorStoreRetriever(vs,
  k=)` wraps any VectorStore/BaseRetriever as Coalent's substrate (artifact id:
  `metadata["artifact_id"]` → `"source"` → `Document.id` → `"id"` → sha1 fallback — give
  documents a `source`); runnable offline refusal-loop example in the LangGraph shape.
- **`SemanticCache.has_source(artifact_id) -> bool`** — True when any cached unit's
  provenance depends on the artifact; the cheap pre-check for change-feed adapters.

---

## What changed in v0.6 (read this before generating code)

The DEFAULT read path is unchanged — a v0.5 setup behaves identically until you opt in
(modulo one bugfix: the v0.5 pool-*preview* `(len, xor-hash)` stale-serve marker hole; if
you never set `serve="pool"` it never affected you). New in 0.6, all opt-in / default-OFF:

- **`read_path="pool"`** (default `"unit"`) — the claim-pool-first read path: every read is
  answered by budget-packing the globally-ranked fresh-claim pool; units remain the
  ownership/freshness/build/provenance skeleton. Hand `result.context["pool"]` to your
  answerer; `result.pool` lists served claims in served order. Measured (n=605, strict
  grading): **0.731 @ 981 mean context tokens** — naive k9's accuracy at ~25% fewer tokens,
  naive's best measured point (k12) at ~43% fewer; ranks p50/p75/p90 = 1/6/15. PARITY at
  fewer tokens, not an accuracy win.
- **`serve_budget`** default split: `None` → **600 unit / 1000 pool** (explicit wins; `<=0` raises).
- **`serve_gate=None`** — pool serve-vs-build: explicit float = absolute (reproducible
  benches); `None` adapts against the pool's null-shaped noise ceiling.
- **`pool_header=None`** → built-in default `## {unit.query[:60]}` per source group
  (fallback `[source: {artifact_id}]`). Measured ladder (n=605, strict): opaque id 0.641 →
  query-title default 0.678 (ships) → caller metadata callable "[title | source | date]"
  **0.731** (the golden path — wire your corpus metadata). Empty return from a set callable
  serves the group HEADERLESS; a raising callable is swallowed.
- **Behavioral stack (all default-OFF):** `residual_spans=True` (build-time span capture of
  extractor-missed fact sentences, on the unit, never in the pool) + refusal fallback
  (`Result.read_id` → `report_refusal(read_id)` returns an attributed retry payload or
  `None`; `report_success(read_id)` confirms) + append-only repair (`lossy_threshold=2`;
  claims never dropped while the source hash is unchanged) + `query_keys=True`
  (`key_floor=0.85`; REQUIRES the pool path — loud `ValueError` otherwise). Measured
  full-loop (n=605, same store): +3.1 pts at +2.1% tokens, refusals 91→61 (−33%), zero
  newly-wrong. Caveats: natural-refusal flip ~20% unconditional (33% when the payload
  contains the gold; 68% lab figure = span-derived questions only); sibling-article key
  collisions in dense same-topic corpora (raise `key_floor`); keys don't move final
  accuracy on diverse rewordings.
- **`reranker=None`** — serving ORDER only; never serve/build/floor decisions.
- **`claim_index=None`** — BYO pool storage (ClaimIndex protocol; numpy/pure-python
  built-ins; per-namespace factory form).
- **New events**: pool_gate, pool_served, pool_masked_stale, pool_budget_overrun,
  rerank_failed, rebuild_triggered_by_read, build_triggered_by_gap, unit_marked_lossy,
  unit_repaired, residual_served, residual_fallback, key_attached, key_confirmed, key_fired.
- **Pool-path Result contract** (unit path untouched): `understanding={"claims":[...]}` (no
  `summary`); `unit_id` = owner of the top served claim; `confidence` = pre-decision pool
  coverage; `evidence=[]` on non-escalated serves (cite via `drill(unit_id)`);
  `recalled=[]`; `cache_hit` = zero synthesis calls this read.
- **Guards (fail loud)**: `query_keys=True` without `read_path="pool"` raises;
  `read_path="pool"` under `HashingEmbedder` raises (use a semantic embedder).
- **DEPRECATED**: `serve="pool"` (v0.5 preview) — verbatim in 0.6, never auto-mapped,
  removal v0.7. Unit-path read knobs are inert on the pool path (warning v0.7, removal v0.8).

---

## What changed in v0.5

v0.5 — "the pool release" — is **all additive, default OFF**; `preset="multi_hop"` arms the
multi-hop set in one argument. New `SemanticCache` parameters (all Since 0.5):

- **`preset="multi_hop"`** — arms cross-unit recall + the hop-2 bridge with calibrated
  thresholds; explicit kwargs override.
- **`widen_chunks=N`** — miss-triggered builds read up to N chunks of the dominant source
  (duck-typed `retriever.widen(artifact_id, limit=)` or a BYO `source_fetcher`) instead of the
  retrieval keyhole. Never fires at ingest. **`widen_on_admission`** (follows `widen_chunks`):
  thin-coverage admission rebuilds widened in place.
- **`provenance_admission=True`** — exact-text containment probe before any build; covered
  reads serve without building (duplicate-understanding prevention).
- **`split_by_artifact=True`** — one unit per source when retrieval mixes artifacts.
- **`adaptive_hit=True`** — the hit gate self-calibrates against cross-unit score inflation as
  the cache grows; repeat/paraphrase queries keep hitting via the seed-reuse channel.
- **`recall_bridge` / `bridge_limit`** — hop-2 bridge restart (rank other units' claims by
  similarity to the matched unit's own claims). Armed by `preset="multi_hop"`.
- **`serve="pool"`** (default `"unit"`) — **experimental** preview of the v0.6 read path: serve
  the token-budgeted, globally-ranked fresh-claim pool; renderers read
  `result.context["pool"]`; stale units' claims are masked the moment a source changes.
  **`serve_budget`** (default 600) is the cost dial; **`pool_header`** is a per-unit header
  line callable ("[title | source | date]").
- **`fast="auto"`** — numpy-accelerated read path (`pip install "coalent[fast]"`), identical
  results (CI-pinned equivalence); pure-Python fallback keeps the zero-dep core.
- **`on_event`** — structured freshness events: unit_built, unit_rebuilt, admission_reuse,
  admission_widen_rebuild, stale_read_prevented, claims_recalled, bridge_claims,
  source_changed.
- **`Result.needs_retrieval`** — hint that coverage fell below the floor.
- **Deprecated: `select_floor`** — superseded by pool serving.

v0.4 recap (still the defaults): **`LLMSynthesizer(extract=True)`** — query-independent atomic
**claims** instead of a question-shaped prose summary (`understanding["claims"]` is the
substance; `summary` may be terse/empty; escape hatch `extract=False`) — and
**`SemanticCache(cross_unit_recall=True)`** — multi-hop bridge facts pooled across all fresh
units at zero extra LLM calls, surfaced as `result.recalled` (escape hatch
`cross_unit_recall=False`).

---

## Recommended setup (v0.6)

```python
# Pool path at the measured operating point, with the behavioral loop wired:
cache = SemanticCache(retriever, synth, embedder=OpenAIEmbedder(),
                      read_path="pool",
                      pool_header=my_header,   # "[title | source | date]" from YOUR metadata (0.731 rung)
                      residual_spans=True, query_keys=True)   # default OFF; the refusal-fallback stack

r = cache.get("what did the report say about Q3 revenue?")
answer = my_llm(r.context["pool"])
if is_refusal(answer):                          # your refusal detector; ONE retry is the measured regime
    retry = cache.report_refusal(r.read_id)     # None when no span clears span_serve_floor
    if retry is not None:
        answer = my_llm(r.context["pool"] + "\n\n" + retry)
        if not is_refusal(answer):
            cache.report_success(r.read_id)     # confirms the rescue -> durable query key

# Unit path (the default, byte-identical v0.5) — the v0.5 recommended block still applies:
# preset="multi_hop", widen_chunks=24, provenance_admission=True, adaptive_hit=True
```

---

## Install

```bash
pip install coalent                 # core, zero required deps
pip install "coalent[openai]"       # OpenAI provider + embeddings (recommended)
pip install "coalent[mcp]"          # the coalent-mcp server (add ,openai for folder mode)
pip install langchain-coalent       # the LangChain integration (its own package)
# extras: anthropic, qdrant, chroma, pgvector, redis, server, langgraph, mcp, fast (numpy), dev
```

---

## Minimal, correct usage

```python
from coalent import SemanticCache, LLMSynthesizer, OpenAIProvider, OpenAIEmbedder, InMemoryRetriever

retriever = InMemoryRetriever()
retriever.add("confluence:hr", "Leave policy: 21 days of annual leave per year.")

cache = SemanticCache(
    retriever,
    LLMSynthesizer(OpenAIProvider(), model="gpt-4o-mini"),   # extract=True by default (v0.4)
    embedder=OpenAIEmbedder(),                               # match by MEANING (recommended)
)

result = cache.get("how much annual leave?")   # the ONE read method
print(result.context["understanding"])         # query-relevant claims + facts
print(result.cache_hit)                        # False (cold) -> True on a similar later query

cache.source_changed("confluence:hr", text="Leave policy: now 25 days.")  # surgical invalidation
# next matching read rebuilds just that one unit, lazily
```

Requires `OPENAI_API_KEY` in the environment for `OpenAIEmbedder` / `OpenAIProvider`. Use
`StubSynthesizer()` (no key) to try the loop offline.

---

## API reference

### `SemanticCache(retriever, synthesizer, *, ...)` — the one read path

All keyword args are optional; defaults shown. v0.4-changed defaults marked ★; new-in-0.5 marked "Since 0.5"; new-in-0.6 marked "Since 0.6".

```python
SemanticCache(
    retriever,                        # a Retriever (below)
    synthesizer,                      # a Synthesizer (below)
    *,
    embedder=None,                    # OpenAIEmbedder() / FunctionEmbedder(fn) / HashingEmbedder(); auto-picks OpenAI when available
    preset=None,                      # Since 0.5. "multi_hop": arms cross-unit recall + hop-2 bridge with calibrated thresholds; explicit kwargs override
    hit_threshold=None,               # match bar; auto-derived per embedder (OpenAI ~0.33, Hashing 0.6)
    hit_margin=0.0,                   # OPT-IN precision guard: refuse a unit that ties a neighbour by < margin
    adaptive_hit=False,               # Since 0.5. hit gate self-calibrates against cross-unit score inflation as the cache grows; repeat/paraphrase queries keep hitting via the seed-reuse channel
    coverage_floor=None,              # escalation bar; auto-derived (OpenAI ~0.28)
    understanding_weight=0.7,         # blend: 0.7*topic(query↔understanding) + 0.3*seed(query↔seed-query)
    route_by_claim=False,             # route by the unit's best CLAIM (late interaction)
    enable_coverage_escalation=True,  # the RAG-floor safety net
    learn_on_escalation=False,        # OPT-IN: cache escalated raw as a new unit (compounding)
    coverage_scorer=None,             # OPT-IN (query, understanding)->float containment check (cross-encoder/NLI/LLM)
    coverage_ceiling=1.0,             # two-tier: consult coverage_scorer only in [coverage_floor, coverage_ceiling)
    relevance_gate=None,              # OPT-IN (query, chunks)->chunks reranker/filter before synthesis
    cross_unit_recall=True,           # ★ multi-hop claim pooling; free on single-hop; auto-off for non-semantic embedder
    recall_threshold=None,            # recall fires when coverage < this (None -> inherits coverage_floor); preset="multi_hop" calibrates it
    recall_limit=6,                   # max claims pooled per read
    recall_bridge=None,               # Since 0.5. None = off unless preset="multi_hop" arms it. hop-2 bridge restart: rank other units' claims by similarity to the matched unit's OWN claims
    bridge_limit=3,                   # Since 0.5. cap on hop-2 bridge claims per read
    widen_chunks=None,                # Since 0.5. miss-triggered builds read up to N chunks of the dominant source (duck-typed retriever.widen(artifact_id, limit=) or source_fetcher) instead of the retrieval keyhole; never fires at ingest
    source_fetcher=None,              # Since 0.5. BYO source-chunk fetcher callable for widening
    widen_on_admission=None,          # Since 0.5. follows widen_chunks: thin-coverage admission rebuilds widened in place
    provenance_admission=False,       # Since 0.5. exact-text containment probe BEFORE any build — covered reads serve without building (duplicate-understanding prevention)
    split_by_artifact=False,          # Since 0.5. one unit per source when retrieval mixes artifacts
    serve="unit",                     # Since 0.5. DEPRECATED in 0.6 (removal v0.7): the v0.5 pool-serving preview; never auto-mapped to read_path="pool". If the pool key is absent (no fresh claims yet), fall back to the standard context
    read_path="unit",                 # Since 0.6. "unit" = byte-identical v0.5 ladder | "pool" = claim-pool-first serving (every read budget-packs the global fresh-claim pool; hand result.context["pool"] to your answerer). Raises under HashingEmbedder
    serve_budget=None,                # Since 0.5; default CHANGED in 0.6: None -> 600 on the unit path (v0.5 preserved) / 1000 on the pool path (the measured operating point). Packed payload tokens, headers counted; <=0 raises
    serve_gate=None,                  # Since 0.6. pool serve-vs-build decision: explicit float = absolute (reproducible benches); None = adaptive against the pool's null-shaped noise ceiling
    pool_header=None,                 # Since 0.5; default CHANGED in 0.6: None -> built-in "## {unit.query[:60]}" per source group (fallback "[source: {artifact_id}]"). Wire your own metadata callable "[title | source | date]" — a measured ~5-pt accuracy ladder. Empty return from a SET callable serves the group headerless; raising callables are swallowed
    reranker=None,                    # Since 0.6. Callable[[str, list[str]], list[float]] — serving ORDER only; never serve/build/floor decisions (a bad reranker can't cause a false serve or skipped build)
    claim_index=None,                 # Since 0.6. BYO pool storage: ClaimIndex instance or per-namespace factory; None -> built-in (numpy / pure-python)
    residual_spans=False,             # Since 0.6. the tier-2 stack: build-time capture of extractor-missed fact sentences as spans ON THE UNIT (never in the pool), side-channel serving, the report_refusal fallback, lossy marking, append-only repair
    span_tau=0.62,                    # Since 0.6. build-time capture bar: fact-bearing sentence w/ max claim cosine below this becomes a span
    span_margin=0.0,                  # Since 0.6. read-time: a span must outrank the best fresh claim by this margin to serve
    span_serve_floor=0.35,            # Since 0.6. report_refusal retry floor (min span-query cosine; up to 2 spans per retry)
    lossy_threshold=2,                # Since 0.6. span serves + raw fallbacks against one unit before lossy-marking (-> append-only repair on next rebuild touch; claims never dropped while the source hash is unchanged)
    query_keys=False,                 # Since 0.6. durable alternate keys from confirmed rescues (report_success). REQUIRES read_path="pool" — raises ValueError otherwise
    key_floor=0.85,                   # Since 0.6. min key-query cosine for a confirmed key to count at all (scores max(content_sim, key_sim)); raise in dense same-topic corpora (sibling-article collisions measured 3/605)
    fast="auto",                      # Since 0.5. numpy-accelerated read path (pip install "coalent[fast]") — identical results (CI-pinned equivalence); pure-Python fallback keeps zero-dep core
    on_event=None,                    # Since 0.5. callable receiving structured freshness events: unit_built, unit_rebuilt, admission_reuse, admission_widen_rebuild, stale_read_prevented, claims_recalled, bridge_claims, source_changed
    select_floor=None,                # DEPRECATED (0.5) — superseded by pool serving. Was: serve atoms by meaning (per-claim cosine >= floor) vs lexical trim
    residual_floor=None,              # OPT-IN (build-time): keep number spans the extractor dropped
    residual_limit=24,
    strategy=ContextStrategy.CONTEXT_FIRST,  # raw only when escalated (default) | CONTEXT_RAW | CONTEXT_ONLY
    store=None,                       # InMemory/SQLite/Redis CognitionStore
    freshness=None,                   # FreshnessPolicy(max_age=, revalidate=)
)
```

Methods:
- `get(query, *, namespace=None, related=3, strategy=None) -> Result` — the read.
- `source_changed(artifact_id, *, text=None, version=None)` — dirty units whose provenance used it (content-hash compare skips no-ops).
- `source_deleted(artifact_id)` — dirty/drop units using it.
- `has_source(artifact_id) -> bool` — Since 0.6.1: True when any cached unit's provenance depends on the artifact — the cheap pre-check for change-feed adapters (skip `source_changed` for never-read files).
- `report_refusal(read_id) -> str | None` — Since 0.6, requires `residual_spans=True`: when YOUR answerer refuses over the served payload, re-scores the namespace's residual spans against the original query embedding and returns an attributed retry payload (or `None` when no span clears `span_serve_floor`). 64-read ring buffer; stale/unknown ids return `None`, never raise.
- `report_success(read_id)` — Since 0.6: the symmetric confirm — call it only when the RETRY answered; confirms provisional query keys durable. Unknown/expired ids are a silent no-op.
- `stats() -> dict` — units, reads, hits, hit_rate, escalations, escalation_rate, active thresholds; Since 0.6 pool additions: pool_serves, probe_reads, admission_reuses, retrievals, rebuilds_by_read, builds_by_gap, serve_gate_effective, pool_noise_ceiling, pool_claims_fresh, pool_claims_total, avg_units_per_serve, pool_mask_rate, pool_scan_slow.

### `Result` (what `get()` returns)

```python
result.cache_hit      # bool — warm semantic hit; on the pool path (Since 0.6): zero synthesis calls ran this read
result.context        # {"understanding": <query-relevant slice>, "raw": [chunks] if escalated/strategy}; on the pool path the packed, attributed payload is result.context["pool"]
result.understanding  # full dict: claims, facts, entities, summary; POOL PATH: {"claims": [...]} with NO summary key
result.evidence       # list[Chunk] — retained raw evidence (the RAG floor); POOL PATH: [] on non-escalated serves — cite via drill(unit_id)
result.coverage       # float 0..1 — how well the matched unit covered THIS query (pool path: final post-build/post-S2)
result.confidence     # match strength; POOL PATH: pre-decision pool coverage (what the gate read) — re-tune any threshold you compare against
result.escalated      # bool — fell back to fresh raw for this query
result.recalled       # list[RecalledClaim(claim, score, unit_id)] — cross-unit claims (v0.4, unit path; [] on the pool path, field kept one version)
result.pool           # Since 0.6, pool path: served claims in served order (RecalledClaim; score = query cosine, unit_id = owner). [] on the unit path
result.read_id        # Since 0.6: deterministic per-read id — the handle for report_refusal / report_success (64-read ring buffer)
result.needs_retrieval# bool — coverage below floor (hint to widen) (Since 0.5)
result.unit_id        # cache unit id; POOL PATH: owner of the top-ranked served claim
result.related        # list[Related] — cross-unit links (shared entity/source)
result.usage          # synth token usage on a miss (None on a hit)
```

### `LLMSynthesizer(provider, *, ...)`

```python
LLMSynthesizer(
    provider,                 # OpenAIProvider() / AnthropicProvider() / StubProvider()
    *,
    model="gpt-4o-mini",
    extract=True,             # ★ query-independent extractive claims (v0.4 default); False = v0.3 prose
    instruction=<default>,    # what understanding to produce; a str or (query)->str. Overrides extract if given.
    fields=None,              # keys to capture (default: summary, claims, entities, facts)
    depth=0.5,                # 0.0 terse … 0.5 balanced … 1.0 exhaustive
    max_tokens=1024, temperature=0.0, retries=1,
)
```
Also: `JSONPassthroughSynthesizer()` (cache structured tool/API JSON with no LLM call), `StubSynthesizer()` (deterministic, offline/tests). `EXTRACTIVE_INSTRUCTION` is exported.

### Retrievers, Embedders, Providers, Stores

- Retriever protocol: `retrieve(query, *, namespace=None) -> list[Chunk]`. `Chunk(artifact_id, text, version="")`; `artifact_id` is the source's natural id (file path, DB PK, doc id, URL) and is what freshness/provenance key on. Shipped: `InMemoryRetriever`, `FunctionRetriever`, `CompositeRetriever`, `BaseVectorRetriever`, `QdrantRetriever`, `ChromaRetriever`, `PgVectorRetriever`.
- Embedders: `OpenAIEmbedder(model="text-embedding-3-small")`, `FunctionEmbedder(fn)` (any local model), `HashingEmbedder()` (zero-dep lexical fallback — recall & semantic coverage auto-disable under it). `default_embedder()` auto-picks OpenAI when `coalent[openai]` + `OPENAI_API_KEY` are present.
- Providers: `OpenAIProvider`, `AnthropicProvider`, `StubProvider`.
- Stores (restart-safe; invalidation graph rebuilds on load): `InMemoryCognitionStore`, `SQLiteCognitionStore(path)`, `RedisCognitionStore(...)`.

---

## The read path — the gate ladder (firing order)

This ladder is the DEFAULT unit path (`read_path="unit"` — byte-identical v0.5). Every
`get()` walks it; defaults are pure cosine (no extra model/dependency).

On the POOL PATH (`read_path="pool"`, Since 0.6) the ladder is replaced: every read ranks
the global fresh-claim pool by query cosine, packs into `serve_budget` under per-source
attribution headers, and serves; `serve_gate` decides serve-vs-build; below
`coverage_floor` attributed raw chunks (`[source: {artifact_id}]`) are appended. Stale
units' claims are masked the moment a source changes. Dedup collapses only a unit's OWN
rephrasings — cross-owner near-duplicates survive as corroboration. The unit-path read
knobs (`hit_threshold`, `hit_margin`, `route_by_claim`, `recall_*`, `select_floor`, ...)
are INERT on the pool path. Freshness/provenance/build semantics are identical on both.

0. **`provenance_admission`** (off; Since 0.5) — pre-build exact-text containment probe: covered reads serve without building.
1. **`hit_threshold`** — best unit's blended score below it -> MISS -> retrieve + synthesize a new unit. **`adaptive_hit`** (Since 0.5) self-calibrates this bar against cross-unit score inflation as the cache grows.
2. **`hit_margin`** (0.0 off) — top beats runner-up by < margin -> ambiguous -> build the query's own unit.
3. **freshness** — dirty/expired -> re-materialize.
4. **coverage** = MAX per-claim cosine of the matched unit.
5. **`cross_unit_recall`** (on) — coverage < `recall_threshold` -> pool best claims across all fresh units (MaxSim); lifts coverage; no LLM call. **`recall_bridge`/`bridge_limit`** (Since 0.5, armed by `preset="multi_hop"`) add a hop-2 restart: rank other units' claims by similarity to the matched unit's own claims.
6. **`coverage_scorer`/S2** (off) — in `[coverage_floor, coverage_ceiling)` a cross-encoder/NLI/LLM check overrides cosine.
7. **`coverage_floor`** — coverage still below -> ESCALATE: append fresh raw retrieval (no LLM call). The RAG floor.
8. **`select_floor`** (off -> lexical trim) — serve atoms by meaning (per-claim cosine >= floor). **Deprecated in 0.5** — superseded by pool serving.
Serving (Since 0.5): **`serve="pool"`** — the v0.5 pool-serving preview; DEPRECATED in 0.6 (removal v0.7), superseded by `read_path="pool"` and never auto-mapped to it.
Build-time: **`residual_floor`** — keep number-bearing spans the extractor dropped. **`widen_chunks`** (Since 0.5) — miss-triggered builds read up to N chunks of the dominant source (never at ingest). **`split_by_artifact`** (Since 0.5) — one unit per source on mixed retrieval.

---

## Why Coalent doesn't serve the wrong answer — Q&A
(Full page: https://coalent.ai/docs/wrong-answers. Unit path and pool path are separate
machines — each answer says which one it describes.)

**Q: How does Coalent know it doesn't have the answer?**
By gating at two different granularities (unit path). The hit gate scores the WHOLE unit —
blended `0.7·topic + 0.3·seed` cosine, floor ~0.33 (OpenAI embedder) — and only answers
"is this the right territory?". Coverage then scores the MAX SINGLE-CLAIM cosine (floor
~0.28) — "does any one claim actually answer this?". Topic similarity is diffuse; fact
similarity is sharp. Worked example: a query "how many sick days do I get?" against a
cached vacation-policy unit clears the hit gate (same HR territory) but NO single claim
mentions sick leave, so the max claim cosine falls below the coverage floor and the read
ESCALATES to attributed raw instead of answering thin. A whole-summary similarity check
would have scored it "covered"; the per-claim check structurally can't.

**Q: But cosine gates aren't perfect — what about near misses?**
Correct, and Coalent's framing is honest about it: the gate is a statistical filter, not a
proof. Near-miss topical serves happen and are SAFE because the payload is attributed
atomic claims, not a prose blob — the answer model sees claims about vacation accrual
under source headers, finds nothing about sick days, and refuses honestly. Measured: 95%
refusal honesty on unanswerable questions vs 85–88% for naive top-k (n=605 news rig).
With the opt-in behavioral stack armed, the refusal then REPAIRS the cache:
`report_refusal(read_id)` → attributed retry payload → `report_success(read_id)` →
durable query key → lossy marking → append-only repair (refusals 91 → 61 = −33%, zero
newly-wrong answers). The stack is the guarantee, not any single check.

**Q: What happens on a coverage failure?**
Escalation — a SERVING action, not a cache mutation: attributed raw retrieval
(`[source: id]`-prefixed chunks, deduplicated, budget-capped, never empty) is appended to
the payload, with no LLM call. The RAG-floor covenant: an escalated read never carries
LESS context than plain RAG would have retrieved. `coverage_floor` (~0.28 OpenAI) keeps
this job on BOTH paths. Coverage failures do NOT trigger rebuilds — the
same-extractor-same-loss finding: if the extractor dropped a fact once, re-running the
same extractor on the same unchanged source mostly reproduces the same loss, so you'd pay
a synthesis call to keep the same gap. Persistent extraction loss is the behavioral
repair loop's job instead (span serves + raw fallbacks → `lossy_threshold` → append-only
repair on the next rebuild touch).

**Q: Rebuild vs escalation — what's the difference?**
They answer different failures. ESCALATION answers a coverage failure (fresh cache,
missing fact): append attributed raw, zero LLM calls, cache untouched. REBUILD answers a
freshness failure (source changed): re-synthesize the unit from fresh source text before
serving — one synthesis call, cache mutated. Triggers: unit path gate 3 (dirty/expired →
re-materialize) and, on the pool path, the P3 freshness mask plus the STALE-OWNED / THIN
probe classes.

**Q: What happens when the pool path's gate says "build"?**
One probe retrieval (the read's single query-shaped retrieval), classified PER ARTIFACT:
CONTAINED (every chunk already retained verbatim by a fresh unit — provenance proves
coverage, zero synthesis; an all-contained probe is a structural admission = duplicate-unit
prevention) · STALE-OWNED (owner unit is stale → rebuild it in place) · THIN (owner unit
under-covers its own source → rebuild in place, widened with `widen_on_admission`) ·
NOVEL (no owner → build a source-anchored unit, one per namespace+artifact). Capped at
≤3 synthesis ops per read, best max-chunk-cosine group first. Empty vs warm cache is the
SAME algorithm with different statistics: empty = everything classifies NOVEL (warm-up,
not a special mode); warm = mostly CONTAINED/STALE-OWNED (a free serve or one surgical
rebuild).

**Q: What is the serve gate threshold?**
The pool path's serve-vs-build decision (stage P4): top PRE-RERANK claim cosine ≥ the
effective gate → serve; below → the probe-classified build read. Semantics:
- `serve_gate=None` (default) = ADAPTIVE: calibrated against the pool's null-shaped noise
  ceiling — the p95 of max claim cosines on provenance-disjoint probes (≤24 fixed-seed
  units), i.e. how high a claim scores on a query it provably cannot answer. Effective
  gate = ceiling + margin, CLAMPED to [coverage-floor default, that + 0.27]
  (~0.28–0.55 on the shipped OpenAI embedder) so a dense near-duplicate pool can never
  gate real answers out.
- Recalibration happens at the END of a build read (post-synthesis — LLM latency hides
  it; the in-flight read used the previous ceiling), at most once per 16 builds.
  Serve-only traffic never pays for calibration.
- An explicit float is ABSOLUTE (adaptation disabled entirely — reproducible benches).
- `stats()["serve_gate_effective"]` is the gate actually applied right now (with
  `stats()["pool_noise_ceiling"]` beside it).
- The decision always reads the pre-rerank cosine — a bad `reranker` can reorder serving
  but can never cause a false serve or a skipped build.
- Reuse bypass: seed cosine ≥ 0.9 (`reuse_threshold`) = the same question asked again →
  forced serve regardless of the gate; a stale reuse match re-materializes in place
  first, so the bypass never serves stale.

**Q: Does the pool path run cross-unit recall?**
No — and this is a common blur. The pool scan IS cross-unit by construction: stage P2
ranks the global fresh-claim pool from ALL units on every read, so there is no anchored
unit to "recall around". Cross-unit recall is the UNIT path's multi-hop mechanism
(Since 0.4, gate 5), firing when the single matched unit under-covers. The `recall_*`
knobs are inert on the pool path and `result.recalled` stays `[]` there.

**Q: What honesty numbers are measured?**
route@1 ≈ 1.00 (right-unit routing, clean structured bench) · misattribution ~0–2% (=
the naive answerer's own noise level; an earlier high reading was a benchmark bug —
contradictory duplicate sources — found, fixed, and documented) · 95% refusal honesty on
unanswerable questions (naive 85–88%) · refusal repair 91 → 61 (−33%) with zero
newly-wrong answers · widened builds read median 23 chunks vs 2 keyhole (rebuild churn
460 → 31). Standing anti-claim: none of this claims accuracy ABOVE naive RAG — the
measured result is parity at fewer tokens; the claim here is that wrong-source, stale,
and fabricated-from-adjacent-context answers are engineered out and honestly bounded.

---

## Which knob for which workload
- `read_path="pool"` (Since 0.6) -> token-efficient serving at naive parity (measured 0.731 @ 981, n=605); wire `pool_header` for per-source questions.
- `residual_spans=True` + wire `report_refusal`/`report_success` (Since 0.6) -> refusal-heavy answer loops / extraction tails — refusals −33% measured, zero newly-wrong.
- `query_keys=True` (Since 0.6) -> repeat traffic with paraphrased revisits — first-pass conversion + fewer fallback round-trips; NOT a final-accuracy lever on diverse rewordings.
- `preset="multi_hop"` (Since 0.5, unit path) -> multi-hop / cross-document questions — arms recall + hop-2 bridge with calibrated thresholds (replaces manual `recall_threshold ≈ 0.7`).
- `widen_chunks` (Since 0.5) -> real-document corpora / cold starts — build from the whole source, not the retrieval keyhole.
- `provenance_admission` + `adaptive_hit` (Since 0.5) -> long-running caches — duplicate-understanding prevention + a hit gate that survives cache growth.
- `on_event` (Since 0.5) -> observability — structured freshness events.
- `hit_margin > 0` -> contradiction/collision-heavy corpora (costs rebuilds; off on clean data).
- `residual_floor` -> messy real prose where extraction may drop a number.
- `coverage_scorer` (S2) -> high-stakes ambiguity (one judge call per borderline read).
- `select_floor` -> deprecated (use `read_path="pool"`).
- `extract=False` + `cross_unit_recall=False` -> exact v0.3 behavior.

---

## Worked examples

**Multi-hop across documents (Since 0.5: one argument):**
```python
cache = SemanticCache(retriever, synth, embedder=OpenAIEmbedder(),
                      preset="multi_hop")   # arms recall + hop-2 bridge, calibrated thresholds
r = cache.get("how does the Growth plan's API limit compare to Enterprise?")
for rc in r.recalled:            # bridge claims pooled from OTHER units
    print(rc.unit_id, rc.claim, rc.score)
```

**Freshness for feed-less API/tool sources:**
```python
from coalent import FreshnessPolicy
cache = SemanticCache(retriever, synth, embedder=OpenAIEmbedder(),
                      freshness=FreshnessPolicy(max_age=3600))   # TTL-refresh
```

**Persistence (restart-safe) + agent/MCP integration:**
```python
from coalent import SQLiteCognitionStore, make_cognition_node, build_mcp_tools
cache = SemanticCache(retriever, synth, embedder=OpenAIEmbedder(), store=SQLiteCognitionStore("coalent.db"))
node  = make_cognition_node(cache)   # LangGraph node: state -> {context: fresh understanding}
tools = build_mcp_tools(cache)       # in-process MCP tool SPECS over cache.get()
```

For a full standalone MCP server (Claude Code / Claude Desktop / Cursor) use `coalent-mcp`
(Since 0.6.1 — see "What changed in v0.6.1" above); for the LangChain-native surface use
`langchain-coalent` (`create_coalent_cache` / `CoalentRetriever`).

**Cache tool/API JSON with no LLM call:**
```python
from coalent import JSONPassthroughSynthesizer
cache = SemanticCache(retriever, JSONPassthroughSynthesizer(), embedder=OpenAIEmbedder())
```

**Events (surgical invalidation from webhooks):** connectors for GitHub / Jira / deploy / CDC + `verify_github_signature`; or call `source_changed(id)` directly. The only contract: the id retrieval stamps as provenance must equal the id the event emits (both derive from the same natural key).

---

## Benchmark (earned, honest)

**Real-world benchmark — v0.6 pool path (news corpus, n=605).** 609 real news articles,
third-party gold questions (the MultiHopRAG corpus); 605 frozen held-out queries; answerer
gpt-4.1-mini; STRICT grading (normalized gold containment in the answer); identical embedder
both arms; naive's own token-scaling curve measured on the same stream.

- **Headline: PARITY with naive RAG at fewer tokens — NOT an accuracy win.** Pool
  **0.731 @ 981 mean context tokens** vs naive k9 0.711 @ 1311 (**~25% fewer tokens**) and
  naive k12 (best measured) 0.731 @ 1729 (**~43% fewer** — 57% of the budget). All CIs
  overlap.
- Naive's own curve, same stream: k4 0.582 @ 590 · k6 0.638 @ 882 · k9 0.711 @ 1311 ·
  k12 0.731 @ 1729.
- **Serving ranks**: gold-claim rank in final pool order p50/p75/p90 = **1/6/15**
  (percentiles over claim-present queries only; `reranker=None`).
- **Behavioral stack**: final 0.769 @ 1003 vs same-store default 0.737 @ 982 = **+3.1 pts
  at +2.1% tokens**; refusals 91 → 61 (**−33%**); zero newly-wrong answers. Same-population
  comparison; requires wiring `report_refusal`/`report_success`.
- **Header ladder** (same frozen store/queries/grader): opaque id 0.641 (153 refusals) →
  shipping query-title default 0.678 (136) → metadata callable **0.731** (101). The 0.731
  headline used the metadata header; the shipping default grades 0.678 on this corpus. The
  gap is a unit-metadata limit (outlet/date live only in corpus metadata), not header format.
- The v0.5 anchor held: 0.699 @ ~1036 (CIs overlap v0.6's 0.731 @ 981). Naive anchors
  reused from the pre-registered sweep, not re-bought. Token counts are packed-context
  tokens, not total round-trip tokens.
- Economics (v0.5 build-layer pilot, same corpus): extractor build spend ~$0.14–0.20 per
  ~600-read stream (gpt-4o-mini); break-even ≈ 4–5 reads/source; widened units read median
  23 chunks vs 2 keyhole; rebuild churn 460 -> 31.
- **Honest limits (part of the claim set):** natural-refusal flip ~20% unconditional / 33%
  when the payload contains the gold (the 68% lab figure holds only for span-derived
  questions) · sibling-article key collisions 3/605 (raise `key_floor` in dense same-topic
  corpora) · keys don't move final accuracy on diverse rewordings · store-build variance is
  real; headline comparisons are same-population by design · build completeness 604/609.
- **Anti-claims:** we do NOT claim to beat naive RAG on accuracy (a pre-registered
  430-token decisive found no beat and no dominance region on the measured curve); we do
  NOT claim compression (naive's own curve reaches the same accuracy given more tokens —
  the value is token efficiency at parity + freshness/provenance + behavioral compounding
  on repeat traffic); we make NO claim about conversational/agent-memory workloads; the
  v0.7 default flip is NOT earned (only the replay gate of five pre-registered gates has
  passed).

**Structured-regime benchmark (synthetic templates).** Reuse workload: 64 sources × 3 seeds =
192 reads/condition, real OpenAI embeddings, deterministic number+attribute accuracy check (no
LLM-judge self-preference), a real dense top-5 retriever shared by both arms (naive RAG *is*
that retriever), accuracy graded escalation-off.

- **Accuracy: parity with naive RAG** across 4 answer models (95% CIs overlap): gpt-4o-mini
  0.81=0.81, gpt-4.1-mini 0.90/0.85, gpt-4o 0.90/0.87, gpt-4.1 0.99/0.97 — at **~47 context
  tokens/read vs naive's 126**.
- **Routing: route@1 ≈ 1.00.** **Misattribution: ~0–2%** (= naive's own answerer noise; an
  earlier high reading was a benchmark bug — contradictory duplicate sources no router can
  resolve — found, fixed, documented).
- **Multi-hop: naive 0% -> Coalent 100%** (cross-unit recall, zero extra LLM calls) — a
  **template-fixture** result; see the real-world benchmark above for uncontrived numbers.
- **Economics: build once (~430 tokens / ~4s per source) -> break-even ≈ 4–5 reads/source.**

---

## Upgrading v0.5 -> v0.6

Nothing changes unless you opt in: the default read path is v0.5 byte-for-byte (one listed
bugfix in the v0.5 pool-preview stale-serve marker). Stores load without migration; v0.6
writes the legacy store format by default, so a rollback to v0.5 reads your store unchanged
(earned spans/keys are simply dropped on a v0.5 load). Two calls now fail loudly instead of
silently misbehaving: `read_path="pool"` under `HashingEmbedder` raises (a keyless v0.5
smoke test that constructs a pool cache may now need an embedder), and `query_keys=True`
without `read_path="pool"` raises. Migrating off the preview: replace `serve="pool"` with
`read_path="pool"` (never auto-mapped) and re-check the pool-path Result contract deltas
above; if you tuned the v0.5 preview, set `serve_budget` to the MEASURED payload tokens
from your `pool_served` logs — the preview under-counted headers. Full guide:
https://github.com/Vectorlink-Labs/coalent/blob/main/UPGRADE-0.5-to-0.6.md

## Upgrading v0.4 -> v0.5

Strictly additive — every new knob defaults off, so bumping the version changes nothing until
you opt in. Recommended: adopt the v0.5 setup block above (`preset="multi_hop"`,
`widen_chunks`, `provenance_admission`, `adaptive_hit`). `select_floor` is deprecated —
superseded by pool serving (`serve="pool"`). Full guide:
https://github.com/Vectorlink-Labs/coalent/blob/main/UPGRADE-0.4-to-0.5.md

## Upgrading v0.3 -> v0.4

Additive except the two default flips. Usually: bump the version, change nothing else (you get
extractive units + multi-hop automatically). If you rendered `understanding["summary"]` as the
answer, read `claims` instead or pass `extract=False`. For exact v0.3 behavior pass
`extract=False` and `cross_unit_recall=False`. Full guide:
https://github.com/Vectorlink-Labs/coalent/blob/main/UPGRADE-0.3-to-0.4.md