What affects crawlability#
- Links: crawlers discover pages by following
<a href>links. Pages with no links pointing to them (orphans) are hard to find. - robots.txt: disallow rules stop compliant bots from fetching URLs.
- Server health: slow responses, timeouts and 5xx errors make search engines crawl less.
- JavaScript: links that only exist after scripts run, or that use click handlers instead of hrefs, may be missed.
- Authentication: content behind logins, paywalls without structured data, or IP blocks.
- Crawl traps: infinite calendars, filter combinations and session IDs that create endless URLs.
Crawlability vs indexability#
Crawlability is whether a bot can fetch a page. Indexability is whether it will store and show it. A page can be crawlable but not indexable (noindex, canonicalised elsewhere, low quality). Both must be right for a page to rank.
Checklist#
- Review robots.txt after every launch or migration.
- Submit a clean XML sitemap.
- Link every important page from at least one relevant page and a hub.
- Fix broken links and redirect chains.
- Run a crawler (Screaming Frog, Sitebulb) monthly and compare with Search Console’s Crawl stats.
Related: crawl budget, Googlebot, indexing.