Geopilot vs a plain prompt

The result

On three projects kept back while geopilot was being tuned, both found every planted bug. Geopilot raised no false alarms and about a fifth as many wrong ideas. A plain prompt raised slightly more useful ideas, and it was cheaper and faster.

On the six projects used for tuning (18 runs each), the plain prompt and geopilot had 0.67 and 0 false alarms, 6.0 and 5.2 useful ideas, and 2.3 and 0.7 wrong ideas.

A false alarm means telling someone to change something that was fine, such as a deliberate noindex on a private page. An idea is any other recommendation, judged by a blind grader that could read the project’s files. It’s useful if it’s specific, accurate and could plausibly help, generic if it would fit almost any site, and wrong if it’s false, reverses a deliberate choice or relies on folklore such as word counts.

On GitHub Copilot, and on a lighter model

We repeated the test on Copilot CLI on the same three projects, with its default model and a lightweight one. Cells show the plain prompt, then geopilot. Copilot doesn’t report a price per run, so cost isn’t compared.

On a lightweight model, geopilot can miss real bugs. On Luna it missed a missing lang attribute in 2 of 3 runs on one project and a button-only link once, and the plain prompt found both every time. By the rule we set in advance, that means geopilot hasn’t shown a win on Luna, even with no false alarms. For an audit, use your host’s default or a stronger model if you can.

Why a plain prompt has more ideas, and why that’s not all good

A plain prompt raises about 15 ideas per report against geopilot’s 11, so it has more to choose from and much more to filter. About 54% of geopilot’s ideas were useful, against 42% for the plain prompt, and the plain prompt raised more than 5 times as many wrong ones. Sorting the useful ideas by kind shows what’s behind the gap of 0.8 per report. The sorting is rough and the samples are small.

• Geopilot raised more ideas about what’s on your pages: 1.8 against 1.0 per report on missing page content, such as facts a page promises but never gives, and 0.9 against 0.4 on links and navigation.
• Geopilot won’t extend a choice you made on purpose. The biggest single difference was naming more training crawlers to block, at 0.7 against 0 per report. Geopilot leaves that decision with you, so you may want to check that list yourself.
• The plain prompt raised more ideas on some basics: titles and descriptions, page structure, canonicals and site settings. This is a real gap in geopilot, and we plan to close it.

Where a plain prompt is still better

• It raises slightly more useful ideas. 6.6 against 5.8 per report, so geopilot missed its target of at least matching it.
• It’s cheaper and faster. Geopilot costs about twice as much and took about 30% longer, and it missed its target of costing less than the previous $0.14. Most of the extra cost is the rulebook it sends to the model, which is also what keeps false alarms down.
• Large repositories are still unproven. In informal runs on one private production repository, geopilot raised specific findings, at about $0.50 and two minutes per run. Its score has varied between runs, and it has sometimes left code unread. It wasn’t graded against an answer key, and we don’t claim geopilot is better on large repositories.

How it was tested

• Criteria first: what would count as a win was written down before any run. The false-alarm result meets it: at most half the plain prompt’s rate, with catch rate and consistency no worse. A later round added targets for ideas, cost and time, and the results above say which were missed.
• Known answers: nine projects on nine stacks, with 38 planted bugs, plus decoys that look like problems but aren’t. Two projects are healthy sites, to catch false alarms.
• Held-out projects: three of the nine were written before the last prompt changes and run only once the prompt was frozen.
• Same conditions: the same model, the same read-only tools and no user settings. Each arm ran three times per project.
• Blind grading: a separate model graded each report against the answer key, with geopilot’s name and scoring lines masked.

Why you should discount this

• Geopilot’s author wrote the projects and the answer key, and the bugs come from geopilot’s own rule areas. That favours geopilot.
• The projects are small. Both arms could read every file, and both scored 100%, so this test can’t show a difference in catch rate. It also says little about large repositories.
• Blinding isn’t perfect. Geopilot’s reports have a recognisable structure even with its name masked.
• Held out doesn’t mean independent. The same author wrote those three projects and knows what geopilot looks for, and three small projects can’t settle this. Geopilot was tuned repeatedly on the other six. Every earlier run is published, and each change is listed in the pre-registration.
• A model judged the ideas. We defined the three classes ourselves, and the grader sometimes gave the same suggestion different verdicts in different reports. Treat small differences in idea counts as noise.
• One measure didn’t work. “Invalid file references” also counted files a report suggested creating, so we don’t report it.
• A few models, one moment. This used Claude Sonnet on Claude Code, and Claude Sonnet 5.5 and GPT-5.6 Luna on Copilot, in November 2026. Other hosts and models weren’t tested.

This measures audit quality only. It says nothing about whether fixes lead to more citations or traffic.

Run it yourself

node evals/run.mjs

This needs an authenticated claude CLI and costs a few dollars. Delete the files in evals/results/ first to rerun from scratch. The projects, answer key, reports and grades are all in the repository.