The geopilot crawler

User-Agent: geopilot/<version> (+https://moelzanaty3.github.io/geopilot/docs/crawler)

• It is a local command-line tool run by a person, not a hosted service. There is no central crawler.
• It reads /robots.txt first and obeys rules for the product token geopilot (or *). A 5xx or unreachable robots.txt is treated as a full disallow.
• It fetches at most two requests at a time per host, 250 ms apart, honours Crawl-delay up to 2 seconds, and stops at 25 pages by default.
• It never impersonates another crawler.
• An operator can override robots.txt only for a target they name on the command line, and that override is recorded in their report.

To opt out, add to your robots.txt:

User-agent: geopilot
Disallow: /