Migration checklist: follow the current migration checklist.
Set a responsible site crawling policy
Choose URL limits, concurrency, delay, response-size limits, rendering rules, and private-network protections before the crawl begins. The policy should be visible in the receipt so another analyst can reproduce the run.
Capture auditable crawler facts
Useful outputs include response chains, canonicals, directives, headings, metadata, links, sitemap membership, structured-data validity, and download size. Raw page bodies rarely need to be retained when extracted facts are sufficient.
Escalate crawl findings selectively
Duplicate content, orphan pages, broken links, canonical conflicts, and sitemap drift can be triaged from crawl evidence. Deep page scans should be reserved for issues that require semantic or competitor context.
A site crawler uses a spider or bot to fetch pages, follow links, and collect data for indexing and accessibility reviews. The results can expose robots directives, performance constraints, and optimization opportunities without treating every scraped page as equally important.
How web crawlers collect and parse information
A web crawler requests a URL, follows the site linking structure, parses text and images, and records response and directive information. Search-engine crawlers such as Googlebot follow robots.txt and related protocols, while agency automation may use a controlled crawler or scraper to inventory websites, domains, features, and accessibility issues for developers and users.