Measured, not promised
How we measured CoreDoc's performance
We took six AI agents — Claude Sonnet 5, Opus 5, Haiku 4.5, GPT Sol, Terra and Luna — and asked each the questions engineering teams actually
prompt AI for: what breaks if I change this, trace this request, explain this function,
plan this feature. Once with the standard toolkit every coding agent has today — read, grep, search. Once
with the CoreDoc map over MCP. Same model, same prompt, same checkout. Every answer was
graded blind against a ground-truth answer key by a judge from the other vendor's family. 437 runs, eight
codebases.
17 100 out of 100
What breaks when a shared type changes — one package, six repos. Without the map the agent found one place in six.
38 100 out of 100
Tracing a request across service boundaries. Without the map, every model stops at the edge of the repo it can see.
½ the tokens · same or better
Where the agent already has the file open, the map replaces the search: 40–60 % fewer tokens on the API-surface and endpoint lanes, up to a quarter less time.
6 of 6 models
Both lanes reproduced on every model we tested, across two vendors. A property of the map, not of a model.
Everything below is the method behind those numbers.
What moved.
Eleven question types, eight codebases, 437 runs — scored 0–100 against the answer key. On the easy ones both arms
hit the ceiling — no story and we don't claim one. The gap lives where the work is hard — wide-reach types,
cross-service traces, code the agent can't even check out — which is exactly where teams currently fall back on
tribal knowledge. Cross-repo questions with no checkout at all are the next section.
The question AI alone + CoreDoc Lift
“What breaks if I change this shared type?” AI alone 17 + CoreDoc 100 Lift +83
“Trace this request across service boundaries” AI alone 38 + CoreDoc 100 Lift +62
“Explain this function” AI alone 67 + CoreDoc 94 Lift +28
“Trace the data flow through this module” AI alone 83 + CoreDoc 100 Lift +17
“Plan this feature” AI alone 86 + CoreDoc 95 Lift +9
“What types does this change touch?” AI alone 94 + CoreDoc 100 Lift +6
“Which component should I reuse?” AI alone 89 + CoreDoc 93 Lift +4
A stronger model doesn't catch up.
29 94
Claude Opus 5 alone vs Claude Haiku 4.5 with CoreDoc, same cross-service trace
38 88
GPT Sol alone vs GPT Luna with CoreDoc, same cross-service trace
The cheapest model with the map beats the strongest without it — the facts live in another repo and no amount of
model strength reads a file the agent never opened.
No repo? Still an answer.
The product team's case: the agent has no checkout — only the map.
Service AI alone + CoreDoc
Trace a request through the gateway AI alone declines + CoreDoc 67
Trace a request through the auth gate AI alone declines + CoreDoc 80
Trace a request through the integrations wrapper AI alone declines + CoreDoc 40
Without a checkout, every baseline run answered in under a thousand tokens: I don't have the code. With the
map, the same agent traced the chain across services it had never opened. Single run per service; repeats are
scheduled and this section updates when they land.
The token bill.
Questions where the agent could already answer from the checkout. The score holds or moves up; the search
disappears — and so does the wait.
Question Tokens Time, s Score
Map the public API surface · Supabase Tokens 1.52M → 858k Time, s 160 → 133 Score 100 → 100
Walk me through this endpoint Tokens 422k → 178k Time, s 94 → 70 Score 100 → 100
Tokens and wall-clock per answer. On the two graph questions above, the map spends more — it's finding code
the agent never would have opened. Here it spends less: one lookup replaces thirty file reads. Same answer, a
fraction of the bill, up to a quarter less waiting.
What we tested on.
Real repos, not benchmarks. Every question has a fixed answer key pinned to a commit.
Codebase
What it is
Scale
PostHog
Open-source product analytics — a monorepo the agent has to navigate cold
~39,000 code files · ~8.7M lines
Supabase
Open-source Postgres platform — TypeScript monorepo, studio app plus APIs
~8,000 code files · ~1M lines
CoreDoc itself
Our own codebase — several services and a shared package across six repos. The cross-service trace, blast radius, no-checkout and feature-plan questions all live here
1,243 files · ~162k lines
The six-model numbers come from CoreDoc's own repos and PostHog. Yes — we tested on ourselves first: it's the
codebase we know to the line, so its answer key is the strictest one we have. Supabase joined the answer-key set on
30 August with one model so far.
How the test works.
Every question runs as a matched pair.
AI alone.
The agent gets the repo checked out and the standard toolkit — read, grep, glob, shell. No CoreDoc, no product
guide. It answers the way it would in your editor today.
AI + CoreDoc.
The same agent, same prompt, same checkout — plus the CoreDoc map over MCP: trace_cross_repo_call, analyze_change_impact, find_dependents and
the rest. Nothing else changes.
Each pair runs three times. Six agents across two vendors: Claude Sonnet 5, Opus 5 and Haiku 4.5; GPT Sol, Terra and
Luna. The judge always comes from the other family — Claude agents are graded by GPT, GPT agents by Claude
— and never sees the source or the map, only the answers and the answer key.
The questions are the ones engineering teams actually ask: trace this request, what breaks if I change this, explain
this function, walk me through this endpoint, where does this data flow, plan this feature. 437 runs across eight
codebases, three access modes — from a full local checkout to no checkout at all.
How answers are scored.
Every answer faces three checks; the score is the answer-key grade with the quote gate — the citation check runs
alongside it.
A script checks the citations.
Every file path, symbol and line the agent cites is verified against the repo at the pinned commit. Wrong path — no credit. No AI involved — this part cannot be charmed.
A blind judge grades the facts.
Each question has an answer key — the required facts a correct answer must contain, plus claims it must not make. The judge marks each fact present, missing or contradicted. One contradicted fact zeroes the answer — the rule is deliberately harsh and it applies to both arms. A confident-sounding wrong answer can’t score well.
A quote check backs every “present”.
For each fact marked present, the judge must quote the sentence that establishes it. If the quote isn’t in the answer, the fact drops to missing.
The checks give 0–100. The number has no literal unit — it's a graded score, not a percentage — but as a
rough guide:
95+ essentially complete
80–95 nearly everything, minor gaps
50–80 enough of the real answer is wrong or missing that an engineer would have to redo the work to trust it
below 50 the answer is worse than not asking
Where the answer key comes from — and why it isn't circular.
The key is built outside the eval, by hand, from the repo at a pinned commit — tracing the request, following
the type, reading the handler. It is never given to the agent and never derived from CoreDoc's output. Both
arms are graded against the same key.
Audit trail
How we keep ourselves honest.
Every run keeps a full audit trail — prompt, agent transcript, answer, citation check, judge verdict with
quotes — failures included. Of 437 runs, four didn't finish: two Haiku runs hit
a turn limit (one per arm) and two map-equipped planning runs on PostHog overran the budget. Failed runs carry no
score — they're excluded and reported separately, never averaged in. Ask us for the bundle in a pilot conversation and trace any number on this page to the run that
produced it.
Where a strong model already has what it needs in the open file, the map's effect is cost and time, not
accuracy — that's the token bill. On PostHog, a strong model with the repo checked out greps its way to the
ceiling on its own; the map's edge there is the bill.
The map's biggest known gap — value-level enum references — was closed in the latest engine build; the
question that exposed it reruns next. Where the agent quoted the map's own summary as a fact and over-claimed,
the judge zeroed it — the same rule we apply to the baseline and the next guide rule on our list. We track what vendors usually hide — DNF counts per arm, run-to-run spread and
the map's own blind spots, because those gaps drive the parser roadmap.
The headline lanes are the latest engine run — one run per cell; the replication claim behind them is the
six-model matrix at three runs per pair. Single-run results — the headline lanes, the no-checkout services, the
feature-plan lane — get their repeats before we lean on them further. Planning answers are graded by the same
blind judge on a rubric while their answer key is finished. In the score table, the shared-type, trace, data-flow
and planning rows are the latest engine runs; the explain-function, type-impact and component rows are
three-run means from the six-model matrix — the endpoint row in the token bill too.
Don't take our numbers. Run this on your own code.
CoreDoc's comparison is built in — answers, speed and spend are tracked per session. In a pilot we run the same
questions with and without the map on your codebase and you read your own numbers, not ours.
CoreDoc