GEO-013 · robots.txt is reachable, parseable and within size limits
Crawlers treat a robots.txt that answers 5xx or cannot be reached as a complete disallow, so a broken robots.txt blocks a whole site.
- Evidence class
- E1
- Weight
- 3
- Subject
- one site subject
- Scope
- URL/repo
- Default severity
- high
- Lifecycle
- stable
- Platforms
- Any web stack
- Readiness cap
- 40, when direct evidence shows robots.txt answered 5xx
Sub-checks
A subject fails if any evaluated sub-check fails, and passes only when every applicable sub-check ran and passed. url sub-checks need a URL; both can also run from source.
| Sub-check | Scope | Checks that |
|---|---|---|
reachability | url | /robots.txt answers 2xx or 4xx, not 5xx or a network error |
parse_size | both | robots.txt is plain text within 500 KiB |
Remediation
Serve robots.txt as text/plain with a 200 (or a 404 if you want no rules). Fix the server error behind a 5xx before anything else.
Fix class: contextual
Known false positives
- A CDN serving a transient 5xx during the scan; re-run to confirm.
Evidence
Platform policies and standards are cited as such; anything else is labelled guidance. These links were carried forward from the specification and should be rechecked before quoting them as current policy.
- RFC 9309: Robots Exclusion Protocol (standard)
Run it
npx geopilot explain GEO-013
npx geopilot scan . --rule GEO-013