grep vs the graph: when Serena's symbols actually beat search

“Plug in an LSP Language Server Protocol — the thing behind 'go to definition' in your IDE. It works off a symbol index, not text. graph (Serena over MCP, in my case) and your agent stops burning context on text scanning,” they said. Symbols instead of lines, precision instead of grep. It sounds like a marketing slogan you’d love to believe. So I didn’t — I measured instead. 23 tasks, four languages.

Spoiler: I later had to retract half of my own conclusions — not because of the tool,
but because of me.

Jump to the results →

TL;DR

I ran the same task twice through two isolated agents Each side is a separate subagent physically given only one toolset: either grep/read/edit, or Serena. Neither knows the other exists, and the task wording is byte-for-byte identical for both. : one with plain grep/read/edit, the other with only Serena’s symbol tools. I measured tokens, time and correctness across four languages: TypeScript, Go, Rust and Kotlin.

Three findings I didn’t see coming:

1. Token savings are bimodal — NOT “a tie”. Where grep is forced to swallow text a regex can’t slice off (multi-line declarations, semantic disambiguation The process of resolving ambiguity, where an algorithm or system pins down the exact meaning of a polysemous word, phrase or term from the surrounding context. ), the graph cuts 12–35% — that’s 5 read tasks out of 15, on three languages out of four. Where grep answers with a single regex, it’s parity. Both extremes are wrong: “Serena always saves context” and “Serena never does”.
2. But the graph’s sturdiest prize isn’t tokens. It’s two other things: correctness where the fact is written nowhere in the code (structural typing), and edit speed (rename/insert up to 65% faster).
3. Half my early findings were killed by measurement bugs, not by the tool. That gets its own chapter.

How it was wired up

For the numbers to mean anything, I had to strip out everything that could lie:

• Isolation. Each agent sees exactly its own toolset.
• Ground truth is built by a real parser (go/ast, the compiler), not a regex, and compared as sets — so “found 3 of 4” can’t quietly become “eh, close enough”.
• Metrics — total tokens, call count and wall-clock, all taken from the subagent’s result.
• Edits run on .git-free copies of the project (why exactly — see the gotchas chapter, there’s a small detective story in there).

What I measured, and on what

Four languages, real projects, not synthetic:

lang	projects	what I could measure
TS	my beloved wife’s app, pallete	struct, refs, def — all
Rust	oswam, vi-book, pallete	impl, refs, rename
Go	my beloved wife’s app	struct, refs, impl + all edits
Kotlin	mo-mo-money, atlas	struct only — the rest unavailable

The Kotlin punchline

Kotlin is the only one of the four where the graph managed to both shit the bed [censored] and be useless (what were you expecting?) [censored] at once.

Shat the bed [censored] : find_referencing_symbols returns an empty {} on Kotlin. So refs, rename, call lookup — anything standing on a reference index — is unbuildable in principle. I first blamed a cold workspace, but the root runs deeper: JetBrains kotlin-lsp doesn’t support KMP Kotlin Multiplatform — shared code across several targets (Android/iOS/JVM). kotlin-lsp doesn't handle it, as stated in its own README. , breaks cross-module references even in an ordinary multi-module project, and its Gradle import returns an empty model (0 libraries to load). No config fixes this. Exactly one tool works — get_symbols_overview — so Kotlin was measured on struct only.

Turned out useless [censored] : Kotlin/Compose files are the thinnest in the corpus (on average 2.5 symbols per file: one-line fun, @Composable). grep takes them with a single regex, while the graph pays a round-trip per file for almost nothing. Across 37 such files that’s grep’s one win outside the noise: +13.5% tokens for the graph. The blunt moral: on a language with thin declarations and no reference index, you don’t need the graph.

Part 1. Navigation: savings where grep READS, not where it greps

Here’s the honest picture across all 15 comparable read runs (both hands warm, both correct — otherwise you can’t compare price), sorted by Δ. Medians, tokens:

task	lang	type	grep	graph	Δ	verdict
API surface (11 files, multi-line interfaces)	TS	struct	45,892	29,742	−35.2%	GRAPH
refs with a 3× Querier collision	Go	refs	40,248	30,189	−25.0%	GRAPH
color-lib (7 files, non-public top-level)	TS	struct	38,728	30,838	−20.4%	GRAPH
impl (structural typing)	Go	impl	31,457	25,682	−18.4%	GRAPH
impl (impl FsOps for RealFs)	Rust	impl	28,296	24,873	−12.1%	GRAPH
service (10 files)	Go	struct	36,301	33,237	−8.4%	tie
refs with a lexical collision	Rust	refs	32,082	30,634	−4.5%	tie
bot (8 files)	Go	struct	31,890	30,563	−4.2%	tie
handlers (39 files)	Go	struct	41,974	41,900	−0.2%	tie
floor probe (1 file)	TS	struct	27,847	27,948	+0.4%	tie
ui-common half (19 files)	Kotlin	struct	33,367	33,591	+0.7%	tie
refs with shadowing (1 symbol)	TS	refs	27,794	28,792	+3.6%	tie
def (5 symbols)	TS	def	24,965	26,241	+5.1%	tie
refs (3 symbols, 3 repeats)	TS	refs	27,251	29,082	+6.7%	tie
ui-common (37 thin files)	Kotlin	struct	34,845	39,553	+13.5%	GREP

Score: 5 graph wins, 9 ties, 1 grep win. (Significance threshold — 2σ ≈ ±10%; anything inside the band is a tie.) Three things only the full table shows:

1. Where the graph saves, it saves big: −12…−35%. Not “a handful of percent” but a third of the read tasks.
2. The win exists on three languages out of four — TS, Go, Rust. Only Kotlin is empty.
3. The driver isn’t the language or the file count, it’s one question: is grep forced to READ text a regex can’t slice off? TS-struct: a naive rg '^export' misses 133 symbols out of 157 (non-public top-level), so grep reads whole files and its cost grows with the text weight. Go/Kotlin-struct: ^func|^type|^fun catches everything in one line per symbol — nothing to read, and nothing for the graph to win on.

The 39-file row (−0.2%) is the instructive one. The graph made 41 calls to grep’s 6 and cost exactly the same. Which gives a rule that saved me a pile of bad conclusions:

Call count is not a proxy for cost. One symbol-overview query ≈ 360 tokens, exactly what grep pays for the same file. Counting calls is only useful as a wandering detector: if there are far more calls than files, the agent is groping.

Part 2. Where the graph wins even at a token tie

The token savings from Part 1 are tied to language and declaration shape — sometimes there, sometimes not. But two advantages hold even where tokens tie.

2.1 The fact isn’t written in the code

In Go a type satisfies an interface structurally A type satisfies an interface automatically if it has the required methods. There's no implements keyword — the link 'type X implements T' is written down literally nowhere. : there’s no implements keyword, and the “type X implements interface T” relationship appears in the source nowhere at all. grep is forced to expand every method of the interface and hand-check each candidate.

Measured on an 8-method interface whose implementation isn’t linked to it by name:

hand	correctness	tokens	calls
grep	4/4 found	31,457	4–9 (method sweep)
graph (find implementations)	3/3	25,682	2

The graph: −18% tokens, −20% time at equal correctness. And here’s the important part — and I mean important: in one early run grep scored recall 0.25 The fraction of truly existing answers a tool actually found. recall 0.25 means it found 1 implementation out of 4. (found 1 of 4). Replays showed that’s stochastic — grep usually copes, it just pays with the sweep. So the claim isn’t “grep can’t”, it’s “grep is flakier and pricier”.

Control check: in Rust, where the implementation is spelled out literally (impl FsOps for RealFs), grep nails it with a single rg. So the blindness is a property of the language, not the task type.

2.2 Edits

Separately I measured Serena’s writing half against boring old Read+Edit:

edit	tool	Δ tokens	Δ time
insert a function	insert_after_symbol	−14%	−65%
replace a body	replace_symbol_body	−7% (tie)	−36%
rename with a collision	rename_symbol	−21%	−57%
bulk replace across 6 files	replace_in_files	+27%	+115%

The first three rows are where addressing a symbol by name throws away the grep → find the window → edit loop. insert_after_symbol doesn’t even read the symbol’s body — the cheapest primitive of the lot. And rename_symbol is structurally safe on top: a name collision (Rendered vs RenderedLine vs rendered) costs it zero, while a manual Edit has to defend every match from accidental clobbering.

replace_in_files, though, is a trap with a bow on it. Serena’s own docs call it the “preferred tool for repeated small edits”, but for an agent with a shell, sed -i does the same thing in one call and twice as fast. A pretty name isn’t a faster one.

A debunked hypothesis. I was sure the edit win would grow with file size — grep supposedly reads the whole file, right? A run on 95/413/744-line files showed a flat baseline cost: a competent model greps the target function and reads the window around it, not the whole file. So on tokens edits are only a tie — the entire win hides in latency.

Fails

Half my early conclusions had to be retracted. And — the important part — every cause was on my side, not Serena’s. Here’s where the harness nearly fooled me:

1. Serena has no directory listing. Symbol overview takes one file, full stop. Without a separate file enumerator the graph wanders through names alphabetically — and that invented a fake +133% and +90%. I handed the graph a Glob for listing and both tasks collapsed back to a tie. Moral: half the graph’s “losses” were my plumbing losing.
2. I never once measured run noise. Mistake: the run’s CV Coefficient of variation — how much a measurement scatters run to run on the same task. 3–5% means any delta under ±10% can't be read as a signal. turned out to be 3–5%, i.e. a 2σ band of ≈ ±10%. Any delta smaller than that I used to read as a signal — that’s how phantoms like “grep is 3.6% better on references” were born. Time is noisier still — CV 30–75%; there you need ≥5 repeats and the median, otherwise you’re measuring CPU weather, not the tool.
3. A git worktree is not a mutation sandbox. A worktree git worktree — a second working directory of the same repository. The catch: they share one .git directory. shares .git, and Serena resolves the project back to the real repo. rename_symbol nearly rewrote my working checkout. Only a .git-free rsync copy saves you.
4. Running the agents in parallel lies about time. They fight over CPU and trigger a Serena startup race that looks like “capability missing”. Run them strictly serially.

What’s next

The writing tools and the structural case were measured on Go — worth replicating on TS and Rust. And I want to test diagnostics on a slow-to-compile language: there get_diagnostics_for_file might win on time too, not just tokens.

Results

The graph is good not in one scenario but in three non-overlapping ones — each for its own reason.

1. When the fact isn’t in the text (structural typing, shadowing, overloads) — the graph is safer and cheaper; grep here either sweeps methods across up to 9 calls or catches the rare recall miss.

2. When you’re editing code — rename is structurally safer (a name collision costs it zero), and insert/replace_body deliver −36…−65% latency. Tokens tie, but time is a real win.

3. When grep is forced to read text a regex can’t slice off — a separate class where the graph cuts tokens, and cuts big: TS-struct with real declarations (−20…−35%), Go-refs with a semantic name collision (−25%), impl on Go and Rust (−12…−18%).

And where it isn’t needed — also specifically, not “in general”: Go/Kotlin-struct on one-line declarations (tie), refs/def by name (tie), Kotlin as a whole (either +13.5%, or unavailable outright). Do bulk replaces with sed, not replace_in_files.

And the discipline that matters more than the choice itself: load exactly one tool per task — ToolSearch The mechanism that loads a tool's schema before its first call. Each schema costs tokens before it does anything useful. on five schemas burns 8 percentage points of the graph’s edge; in struct always give it a file enumerator; and don’t judge cost by call count.

So, in one line: the graph isn’t “cheaper in general” — it’s more precise where the text stays silent, faster where you’re writing, and cheaper where grep is forced to read. Everything else is marketing. 🖖