Crawlability screen
Crawlability answers a different question from Visibility: not whether assistants mention you, but whether the crawlers that read your site are allowed to, and whether they can actually get through.
Silktide checks the 's own web addresses. You do not need a linked website or Analytics for this screen to work.
What it checks
For each web address on your identity brand, Presence reads robots.txt and llms.txt, then probes the crawlers Silktide tracks:
- Search engines such as Googlebot, Bingbot, and Applebot
- AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, and CCBot
Blocking a search crawler is more serious than blocking an AI bot: it can remove the site from that engine, not just from AI answers.
Declared vs effective access
Two findings are kept separate:
- Blocked in robots.txt means the site's published rules say no.
- Blocked in practice means the server refused the crawler even though robots.txt allowed it. That often points to a firewall or CDN rule nobody remembers configuring.
When Silktide cannot send a live request as a crawler, the screen shows what robots.txt declares without claiming a live probe happened.
When everything says "Not assessed"
Every check starts with a control: Silktide fetches the page as an ordinary web browser. If the site turns even that away, no verdict can be pinned on any individual crawler, so the whole column shows Not assessed instead.
This usually means the site sits behind a bot-protection service (such as Cloudflare) that challenges automated visitors it does not recognise, Silktide's checks included. It does not mean crawlers are blocked. Major search and AI crawlers prove who they are by their network address, and bot-protection services routinely let those verified crawlers through while refusing everyone else. Silktide cannot borrow those addresses, so it reports what it could not see rather than guessing.
A quick way to reassure yourself: if the site's pages appear in Google's results or get cited by AI assistants, real crawlers are reading it.
Schedule and Probe now
Workspaces are checked on a weekly schedule so outbound requests stay polite. Probe now runs a fresh check when you have just changed robots.txt and do not want to wait.
Recent rule changes appear in a Changes list for the last few weeks.
Competitors
When you track competitors, the same grid shows their web addresses beside yours so you can see whether a rival is open to crawlers you have blocked, or the other way around.
Related
- Analytics - visits those crawlers and platforms send after they can reach you
- AEO reports - per-page readiness judgements, including crawler access
- Presence and AEO - readiness, visibility, and referred traffic