When crawling fails
When a website will not start, finish, or return the pages you expect, work through these causes in order. Many "empty inventory" problems are crawl problems, not scoring problems.
Confirm the address
Small URL mistakes are common:
www.present or missinghttpvshttps- A staging host that only resolves on your network
- A path that redirects somewhere else
Silktide tries to detect and guide you around many of these, but it cannot invent a URL that does not exist.
Bot protection and firewalls
Many sites sit behind Cloudflare or similar anti-bot layers. Silktide loads pages in a real browser, but those defenses can still block or challenge the crawler.
Set a proxy under Advanced settings so traffic comes from known Silktide IPs. See Where Silktide connects from. If the site rate-limits testing, lower max connections before anything else.
Content that differs from your browser
If Silktide sees different language, currency, homepage, or missing sections:
- The site may vary by geography - pick a proxy close to the audience you care about
- Personalization or A/B tests may depend on cookies you have but the crawler does not
- Bot protection may be serving a challenge page instead of the real site
More detail: Why is Silktide showing different content than my browser?.
Authentication
Password-protected sites need credentials. HTTP authentication is configured in Advanced settings; form-based login needs a configured login flow. Silktide cannot complete CAPTCHAs or multi-factor codes. See Can Silktide test password-protected websites?.
Scope and discovery gaps
If the site tests but important pages are missing:
- Check allowed, denied, and forced rules
- Confirm links, sitemaps, and forced pages under Discovery
- Review Inventory coverage and, where Analytics is connected, Untested pages
Silktide does not treat as a hard limit by default - see Does Silktide respect robots.txt?.
Failed downloads
Pages Silktide could not fetch appear under Inventory > Failed downloads, with a status such as Rate limited and the HTTP code. That distinguishes unreachable pages from pages that do not exist.
Ask for help
If Silktide AI is enabled, ask the Agent to help diagnose the failure. Otherwise contact Silktide support with the website URL and what you see (errors, partial crawl, or empty inventory).