release: v1.9.0 — evidence-driven ranking, half the query latency - #25
Merged
Conversation
…n half
RRF fused on (path, line-bucket), so a symbol definition at line 40 and a
lexical hit at line 120 in the same file produced two weakly-scored candidates
instead of one strong one — two retrievers agreeing on a file never reached the
score at all. Each candidate now also receives, at file_agreement_weight (0.4),
the RRF mass of every retriever that found its file at another locator,
excluding retrievers already counted at that locator so nothing double-counts.
Deepen the documentation prior from -0.05 to -0.20: prose describing a feature
matches a natural-language question more literally than the code implementing
it, so design notes were displacing the modules they describe. Because this only
reorders prose relative to code, documentation-seeking queries improved too
(category MRR 0.579 -> 0.612); -0.35 overshoots and collapses them. Generated
and vendored paths move to -0.25 to stay the least-preferred role, and
MAX_ABS_PRIOR pins the invariant that priors remain tiebreakers.
Fuzzy identifier matching becomes a recall fallback, running only when the
precise lookup named no symbol and returned fewer than fuzzy_fallback_min rows.
It moved no ranking metric across 305 queries while accounting for ~20% of query
latency; typo and acronym recall is unchanged because those are exactly the
queries where the precise lookup comes up empty.
Make candidate over-fetch explicit as candidate_pool_multiplier instead of an
implicit side effect of enabling dedup or MMR. That alone showed the quality
previously credited to SimHash dedup was really the wider pool; dedup is kept
for what it measurably does, which is cutting the duplicate rate of returned
snippets from ~1.6% to ~0%. SimHash itself now folds repeated tokens by
multiplicity and caches token digests, bit-for-bit identical but ~2x faster.
Also fixes two latent bugs: exactness was read from SQL relative to whichever
needle retrieved the row, so a synonym expansion ("config" for "configuration")
marked an unrelated symbol as an exact match, worth +0.20 at rerank and an
unconditional high confidence; and duplicate suppression broke ties with ">",
handing the slot to whichever equal-scoring copy arrived last and contradicting
its own documented "ties favor input order".
Measured against 1.8.0 over 305 queries on Python, Java and TypeScript corpora:
MRR +0.027, MAP +0.028, nDCG@10 +0.024, recall@5 +0.031 (all p < 0.001), with
p50 latency 78.6ms -> 51.2ms and p95 193.2ms -> 94.9ms.
The benchmark was 36 hand-written queries against this repository in Python. At that size a +-0.03 MRR move is a single query, so a delta column could not tell an improvement from a reshuffle, and nothing guarded against tuning the ranker to one repository in one language. gen_queries.py mints objective ground truth from any git repository by pairing a human-written commit subject with the files that commit actually changed. Unlike docstring-derived benchmarks the query text lives in git metadata rather than in the indexed corpus, so the answer is not copied into the document being retrieved. Only localised, described changes survive: no merges, reverts, releases, version bumps or formatting commits; at most --max-files files, all of which must still exist at HEAD; duplicate subjects collapse. Changelog-style files are never accepted as answers because they paraphrase subjects, and any commit touching tests/eval is dropped so the set cannot grade itself. run_eval.py gains --corpus REPO:QUERIES to pool several repositories into one benchmark, weighting every query equally rather than averaging per-corpus averages. Every non-baseline row now reports a seeded paired bootstrap 95% confidence interval and a paired permutation p-value, so signals ship on the strength of that test rather than the sign of a delta. The report also tracks agent-facing noise: mean emitted snippet tokens (results past the budget carry no snippet and are not billed), duplicate rate of returned results, and p99 latency.
Ranking quality and latency both improve against 1.8.0, measured over 305 queries on Python, Java and TypeScript corpora: MRR 0.564 -> 0.591, MAP 0.433 -> 0.461, nDCG@10 0.503 -> 0.527, recall@5 0.529 -> 0.560 (all p < 0.001), emitted tokens 1183 -> 1091, p50 78.6ms -> 51.2ms, p99 277.1ms -> 152.2ms. Documentation now describes the shipped implementation rather than the plan. RETRIEVAL.md gains the cross-locator agreement formula and a source-prior table, and drops a "recency" rerank feature that was listed but never implemented. BENCHMARKS.md adds the retrieval eval as a fourth surface and is explicit that it measures version-over-version deltas, not absolute superiority over any other tool.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The retrieval benchmark was 36 hand-written queries against this repository, in
Python. At that size a ±0.03 MRR move is one query, so a delta column could not
separate a real improvement from a reshuffle, and nothing guarded against tuning
the ranker to a single repository in a single language.
Three defects were found once the measurement was strong enough to see them:
(path, line-bucket). A symbol definition at line 40 and a lexical hit atline 120 in the same file land in different buckets, so a file that two
retrievers agreed on fused as two weak candidates rather than one strong one.
agreeing_sourceswas computed but only used for confidence.natural-language question more literally than the code implementing it.
was ~20% of query time; the quality credited to SimHash dedup was actually
coming from the pool widening that enabling dedup implicitly caused.
Change
tests/eval/gen_queries.pyderives objective, leak-freeground truth from any git repository (commit subject → files that commit
changed; the query text is git metadata, not corpus content).
run_eval.py --corpus REPO:QUERIESpools repositories, and every non-baselinerow carries a seeded paired bootstrap CI and permutation p-value.
file_agreement_weight(0.4), with theRRF mass of retrievers that found its file at another locator, excluding
retrievers already counted at that locator.
least-preferred role;
MAX_ABS_PRIORpins the tiebreaker invariant.fuzzy_fallback_min).candidate_pool_multiplierreplaces the implicit over-fetch.bit-for-bit identical, ~2× faster.
Fixes two latent bugs: synonym expansions were being reported as exact symbol
matches (+0.20 rerank, unconditional
highconfidence), and duplicatesuppression broke ties in the opposite direction to its own docstring.
Results
Against v1.8.0 over 305 queries (Python, Java, TypeScript), identical harness,
old source checked out in a worktree:
Rejected by measurement
Kept out of the default because the data did not support them: IDF-weighted
lexical coverage (significantly worse, p=.011), stem-prefix expansion,
IDF posting-scan candidate generation, a larger FTS pool, score-aware fusion,
intent weight retuning, and a file-length prior — the last looked compelling
descriptively (Java top-1 results averaged 124.5 chunks vs 8.6 for ground truth)
but the medians matched, the mean was outlier-driven, and it was significantly
harmful at every strength tried.
An oracle bound explains why every recall-side idea failed: perfectly reranking
the current candidate pool would give MRR 0.905 vs 0.593 actual, so the
remaining loss is ranking (0.312), not candidate generation (0.095).
Verification
ruff check src testsclean;mypyclean across 60 filespython -m build,twine check, andscripts/release_smoke.pyall passCompatibility
No CLI, MCP, Skill, or API surface changes. Every new signal sits behind an
independent
RetrievalTuningflag, andRetrievalTuning.baseline()stillreproduces the pre-1.8.0 pipeline so the ablation baseline does not drift.