If you found this page from a User-Agent string in your access logs, you're in the right place. This page explains who operates ScryBot, why it might be crawling your site, what it collects, and how to control or stop it.
ScryBot is the crawler operated by Scry (scrystudio.com), a content discovery and migration tool. It is not a search engine crawler and not a general-purpose scraper — every crawl starts from a site a Scry customer has explicitly configured as a target, as part of building a content inventory for a website migration or governance project. (One nuance on scope: a page ScryBot crawls can reference files or images hosted elsewhere — a third-party CDN, for example — and those get fetched too, the same way a browser loading that page would. See "What does it collect?" below.)
A Scry customer added your site's URL as a crawl target for their own migration or content-inventory project — typically because your organization (or a team within it) is the customer, replatforming or auditing this same site. ScryBot never discovers or crawls a site on its own initiative; every crawl starts from a target URL a customer typed in.
For each page it fetches, ScryBot records:
<a href>) and embedded (<img>) — are fetched and registered: URL, filename, content type, byte size, and a content hash (used to detect duplicate files), plus an image's alt text when present.ScryBot does not submit forms, execute arbitrary JavaScript on your pages, or attempt to log in or access anything behind authentication.
By default, every crawl respects your site's robots.txt for the pages it visits. To block ScryBot from crawling pages entirely, add:
User-agent: ScryBot Disallow: /
You can also scope that to specific paths (e.g. Disallow: /internal/) instead of blocking the whole site. ScryBot checks your robots.txt before every page it fetches, working from a copy it re-fetches at most once an hour — so a rule you add now takes effect within the hour in the ordinary case.
Two honest caveats, rather than a blanket guarantee: robots.txt governs which pages get crawled — files and images a crawled page references (PDFs, documents, embedded images) may still be fetched from wherever they're hosted, including a third-party CDN, even if that host's own robots.txt disallows it; and on rare crawls a customer has explicitly configured to override robots.txt, a Disallow rule won't stop the crawl on its own. If you add a rule and still see ScryBot traffic past the hour, email us (below) — that is the reliable way to get it stopped, and we treat it as urgent.
ScryBot paces its requests to your site with a configurable per-host delay between requests — one second between requests by default, never faster than 250ms — and a cap on how many requests it makes to your host at once. If ScryBot's traffic is heavier than you'd like, email us (below) with your domain and we'll ask the customer running the crawl to increase the delay — or, if it's more urgent, add a Disallow rule as above, which stops page crawling on your side within the hour in the ordinary case.
ScryBot identifies itself with the following User-Agent string on every request, so you can match it exactly against your logs:
ScryBot/1.0 (+https://scrystudio.com/bot)
Questions, abuse reports, or slow-down requests: akempler@gmail.com.