Crawling

Before Silktide can test anything, it has to fetch it. A starts from your website's home page, reads what it finds, follows links, and keeps going until it has covered everything in scope.

Most of the time this is invisible and you only see the results in Inventory. This area explains how that fetch works - and what to do when a website answers slowly, refuses requests, or serves something different to Silktide than it does to you.

How crawling works

How Silktide crawls - real browser testing, pages vs files, and the overall fetch model.
Discovery - how URLs enter the queue from links, sitemaps, and forced pages.
Scope rules - allowed, denied, and forced rules, and why the whole website is URL-only.

Connecting and loading

Where Silktide connects from - AWS IPs, proxies, and the user agent.
Loading a page - timeouts, cookie banners, scrolling, stealth, and other Advanced settings.

When something goes wrong

Rate limiting and retries - what happens when a site refuses requests that arrive too quickly.
When crawling fails - bot protection, authentication, wrong content, and other common blockers.

Testing scope - how Silktide decides which content to discover and analyze.
What we test - configure allowed, denied, and forced URL rules.
Advanced settings - parallel requests, page timeouts, proxy, and other loading controls.
Inventory coverage - why what Silktide found may differ from a list.