Ranked grep, measured against everything else

1 · One sentence

You type a word into a code-search tool. How far down the list is the thing you were looking for? Everything on this page is the consequence of trying to answer that honestly.

2 · The amnesiac librarian

Imagine a librarian who is fast, tireless, and has complete amnesia every morning. Ask her “where do we handle refunds?” and she walks the shelves looking for the word refund, then hands you every matching book — in shelf order. Not “most useful first”. Shelf order.

That librarian is grep, and every tool like it. Not stupid — prior-free: nothing ever gave it an opinion about which result is likelier to be the one you wanted. A coding agent reads the first page of results and acts on what it sees, so for agents the position of the answer is not cosmetic. It is the whole product.

hay is the same librarian with exactly one added opinion: a book that defines a thing is probably more useful than a book that merely mentions it. This benchmark asks whether that one opinion is worth anything.

3 · Grading without self-grading

To score an answer you must already know the right answer, for every question — normally a funded project with human annotators. The tempting wrong idea is to ask hay what the answer is: the tool would grade its own homework. Instead the answers come from a parser (ast-grep reads source like a compiler): which symbols are declared exactly once in this repository, and in which file? Different mechanism from any tool being tested, so no circularity — and “exactly once” because a symbol declared in three places has no single right answer at all.

4 · Turning position into a number

Averaging raw positions is wrong, because the gap between 1st and 2nd is enormous and the gap between 50th and 51st is nothing. So flip it — score by 1 divided by the position:

Average that across queries and you have MRR — mean reciprocal rank. It falls exactly the way the experience falls.

5 · Same questions, then shuffle

Every tool runs on identical queries, so hard questions cancel out — you weigh yourself on the same scale before and after. For an invocation that guarantees stable order, each per-query difference is then resampled ten thousand times (the bootstrap), and Fisher's randomization test asks the same question by flipping signs at random. A difference is claimed only when both methods agree. Tools without a stable-order contract remain visible as descriptive snapshots; running statistics over a scheduler accident would manufacture precision.

6 · Two different ways to win

Searching ripgrep's own source for quiet: plain ripgrep returns 73 lines, hay returns 72, ast-grep returns 11 — and it is right to throw them away, because it parses the code and knows a comment or a string is not a use of the symbol. So: ast-grep filters, shrinking the list and keeping walk order; hay ranks, keeping everything but sorting the declaration first. Same median, different failure mode — when luck turns, ast-grep drops to 5th while hay is still 1st. That last part is a hypothesis, flagged as such, not a measured finding.

7 · The bug worth more than the benchmark

hay once scored below ripgrep on the Linux kernel. The mechanism was embarrassingly simple: hay spots declarations by keywords like fn, function, def — and C has none of them. Its core feature had never worked on C, proven by switching it off and watching the kernel score not move. Building the honest test found the bug. The fix is in every result below.