Does Silktide respect robots.txt?

By default, no. Silktide purposely does not treat robots.txt (or typical noindex / nofollow hints) as hard limits on what it may test.

Search engines use those files to decide what to index. Silktide's job is different: find problems that real people can hit, including pages that robots.txt happens to block. Many robots.txt files accidentally hide user-facing content from ordinary crawlers.

You still control what Silktide tests:

• Include and exclude rules under What we test
• Page allowance and PDF limits
• Crawl speed and parallel request settings in Advanced settings

If you specifically need robots.txt rules enforced for a site, contact Silktide support - that can be configured when there is a clear reason.