SiteOne Crawler

SiteOne Crawler is a single-binary website crawler and analyzer for Windows, macOS, and Linux, with no runtime dependencies. Rewritten in Rust, it crawls every page and asset, follows robots.txt, and reports status codes, response times, sizes, and errors like 404s and redirects.

Built-in analyzers cover SEO, security headers, accessibility, best practices, performance, and SSL/TLS, with a quality score from 0.0 to 10.0 across five categories. Results come as an interactive HTML report, structured JSON, or colored terminal text, and can be emailed through a built-in SMTP sender.

It also clones whole sites to browsable offline files, converts pages to clean Markdown for feeding to AI tools, and generates sitemap.xml and sitemap.txt. A CI/CD quality gate with configurable thresholds exits with code 10 on failure, and the crawler doubles as a stress tester and cache warmer.