If you found this page from a User-Agent string in your access logs, you're in the right place. This page explains who operates ScryBot, why it might be crawling your site, what it collects, and how to control or stop it.
ScryBot is the crawler operated by Scry (scrystudio.com), a content discovery and migration tool. It is not a search engine crawler and not a general-purpose scraper — every crawl starts from a site a Scry customer has explicitly configured as a target, as part of building a content inventory for a website migration or governance project. (One nuance on scope: a page ScryBot crawls can reference images or files hosted elsewhere. Images are only fetched from a host that's part of the crawl target or one the customer has told Scry they control; linked files follow the same rule — the crawl's target scope, or a host the customer has approved. A link within the crawl's target scope is crawled like any other in-scope page; a link to an approved host outside that scope is only ever catalogued as a file there, never crawled as a site. And stylesheets (only on crawls with page archiving turned on) are fetched from wherever they're hosted so an archived page keeps its layout. See "What does it collect?" below for the full breakdown.)
A Scry customer added your site's URL as a crawl target for their own migration or content-inventory project — typically because your organization (or a team within it) is the customer, replatforming or auditing this same site. ScryBot never discovers or crawls a site on its own initiative; every crawl starts from a target URL a customer typed in.
For each page it fetches, ScryBot records:
<a href>), embedded (<img>), and stylesheets — are recorded: URL, filename, content type, byte size, and a content hash (used to detect duplicate files), plus an image's alt text when present. But the three follow different rules for which host they're actually fetched from, not just recorded:robots.txt for the referenced path before fetching. An image referenced at any other host is recorded by URL only and never fetched.<a href>) are followed within the crawl's target scope, or at a host you've told us is yours — and are checked against that host's robots.txt at the URL your page references. If your site redirects that link to a storage host — a CDN or cloud bucket — ScryBot follows the redirect the way a browser would, the same as any other link. A link to an approved host is only ever recorded as a file: if it turns out to be an HTML page rather than a document, ScryBot fetches it once, records nothing from it, and does not crawl further links from that host — an approved host is never treated as a site to crawl.robots.txt, per-host pacing, and a cap on how many distinct third-party hosts one crawl will reach out to.ScryBot does not submit forms, execute arbitrary JavaScript on your pages, or attempt to log in or access anything behind authentication.
By default, every crawl respects your site's robots.txt for the pages it visits. To block ScryBot from crawling pages entirely, add:
User-agent: ScryBot Disallow: /
You can also scope that to specific paths (e.g. Disallow: /internal/) instead of blocking the whole site. ScryBot checks your robots.txt before every page it fetches, working from a copy it re-fetches at most once an hour — so a rule you add now takes effect within the hour in the ordinary case.
An honest breakdown, rather than one blanket guarantee — robots.txt governs which pages get crawled, and it also governs the other three fetch paths, but each checks a different host's file:
robots.txt on the host it's referenced at — but only when that host is part of the crawl target or one you've told us you control. An image at any other host is never fetched at all, so its robots.txt is moot.robots.txt at the URL your page links to — if that redirects to a storage host, the storage host's own robots.txt doesn't get a veto, the same way a browser doesn't ask permission mid-redirect.robots.txt on whichever host actually serves it, even a third-party CDN your site didn't configure as a crawl target — deliberately different from the other two, because a page's layout usually depends on CSS hosted somewhere else entirely.One more honest caveat: on rare crawls a customer has explicitly configured to override robots.txt, a Disallow rule won't stop the crawl on its own — though it never applies to a host you haven't claimed as yours, since ScryBot doesn't fetch from those hosts regardless. If you add a rule and still see ScryBot traffic past the hour, email us (below) — that is the reliable way to get it stopped, and we treat it as urgent.
ScryBot paces its requests to your site with a configurable per-host delay between requests — one second between requests by default, never faster than 250ms — and a cap on how many requests it makes to your host at once. If ScryBot's traffic is heavier than you'd like, email us (below) with your domain and we'll ask the customer running the crawl to increase the delay — or, if it's more urgent, add a Disallow rule as above, which stops page crawling on your side within the hour in the ordinary case.
ScryBot identifies itself with the following User-Agent string on every request, so you can match it exactly against your logs:
ScryBot/1.0 (+https://scrystudio.com/bot)
Questions, abuse reports, or slow-down requests: akempler@gmail.com.