GEO-009 · Search crawlers have the access your declared policy says they should
Compares effective RFC 9309 access for each search crawler with the owner-declared policy. Training crawlers are reported for information only.
- Evidence class
- E1
- Weight
- 3
- Subject
- one site subject
- Scope
- URL/repo
- Default severity
- high
- Lifecycle
- stable
- Platforms
- Any web stack
- Readiness cap
- 40, when direct evidence shows a search crawler is blocked site-wide against declared policy
Sub-checks
A subject fails if any evaluated sub-check fails, and passes only when every applicable sub-check ran and passed. url sub-checks need a URL; both can also run from source.
| Sub-check | Scope | Checks that |
|---|---|---|
search_access | both | every search crawler is allowed (or blocked) on sampled paths as policy.aiSearchCrawlers declares |
Preconditions
- policy.aiSearchCrawlers is allow or block; undecided makes the rule not applicable
Remediation
Adjust the user-agent groups in robots.txt so the crawlers your policy allows can reach the paths you want found. Crawler-specific fixes stay manual until the Phase 2 equivalence proof ships.
Fix class: guarded (Phase 2)
G5Phase 2, not in this release
Known false positives
- A user-agent group written for a crawler token GEO does not list.
Evidence
Platform policies and standards are cited as such; anything else is labelled guidance. These links were carried forward from the specification and should be rechecked before quoting them as current policy.
- RFC 9309: Robots Exclusion Protocol (standard)
- OpenAI crawlers (platform-policy)
- Anthropic crawlers (platform-policy)
- Perplexity crawlers (platform-policy)
- Google: AI features and your website (platform-policy)
Run it
npx geopilot explain GEO-009
npx geopilot scan . --rule GEO-009