Crawler

A crawler is a program that reads websites automatically by following links from page to page. Search engines use crawlers to build their indexes, to read the web, and Silktide uses one to discover and test the pages of your website.

Silktide's crawler loads pages in a real browser rather than only reading raw HTML - see How Silktide crawls. Sites control crawlers with and firewall rules, which is why a page that works in your browser can still be invisible to a crawler.