Rate limiting and retries

Some websites deliberately refuse requests that arrive too quickly. This is called rate limiting, and it is usually enforced by a CDN, firewall, or hosting platform rather than by the website itself.

Testing reads a lot of pages in a short time, so it is one of the most common things to trigger a limit. The pages are fine; the server is simply declining to serve them that fast.

What Silktide treats as rate limiting

Two responses are read as "slow down":

• 429 Too Many Requests - the standard way a server says you are asking for too much.
• 503 Service Unavailable, when it includes a Retry-After header - some platforms use this instead. Without Retry-After, a 503 is treated as a genuine outage rather than a limit, because that is usually what it means.

What happens next

When Silktide sees one of these responses, it stops requesting anything from that website for a while - not just the page that was refused. Limits almost always apply to the whole domain, so continuing to request other pages would keep the limit active.

Silktide then tries the refused page again. If it is refused a second time, the wait before the next attempt is longer, and longer again after that. Backing off in increasing steps gives the limit time to reset instead of repeatedly walking into it.

If the response includes a Retry-After header, Silktide waits for the period the server asked for. There is an upper bound on this, so a very long Retry-After will not hold up the rest of your test indefinitely.

Silktide also slows down for the rest of that test. Each time the website refuses a burst of requests, Silktide halves how many pages it fetches from it at once, down to one at a time. Your max connections setting is not changed, and the next test starts at the full number again.

When Silktide stops retrying

Retries are not unlimited. If a page is still being refused after several attempts, Silktide records it as Rate limited and moves on so the test can finish.

While the website keeps answering some requests, Silktide keeps going, one page at a time if it has to, however long that takes. It stops asking the website for the rest of a test only when the website answers nothing at all in between: after 30 refused requests in a row, or three pages that used up every retry. The pages it did not get to are recorded as Rate limited. Time spent waiting does not count, only refusals.

Pages in this state appear under Inventory > Failed downloads, with the status and the HTTP code that was returned. That distinguishes a page Silktide could not reach from a page that does not exist - the two look identical if you only compare page counts between tests.

Why this matters for your scores

A page that could not be fetched cannot be tested, so it contributes nothing to your scores. If a different set of pages is refused each time you test, page counts and scores can move between tests even though nothing on the website changed. See Why did my score change when the website did not? for the other common causes.

What you can do

If a website is being rate limited regularly, you have three options, roughly in order of how easy they are:

1. Lower parallel requests. Advanced settings controls how many pages Silktide fetches at once for that website. Reducing it makes testing slower but far less likely to trip a limit. This is the first thing to try.
2. Allow Silktide through. Ask whoever manages your CDN, firewall, or hosting to exempt Silktide from rate limiting. See Where Silktide connects from for the details they will need.
3. Test less at once. A smaller page allowance or a narrower testing scope reduces the total volume of requests.

Where a website is managed by a third-party platform, the limit is often set by that platform rather than by your own infrastructure, and exempting Silktide may need to go through them.