What is crawl budget?#
Google describes crawl budget as a combination of crawl capacity limit — how fast it can crawl without overloading your server — and crawl demand — how popular and fresh your URLs are. Together they decide how many of your pages Googlebot fetches and how often.
Does your site need to worry?#
For most small and medium sites (a few thousand URLs), crawl budget isn’t a problem; Google crawls them efficiently. It becomes important for:
- Large sites with tens of thousands of pages or more.
- E-commerce sites with faceted navigation generating endless URL combinations.
- Sites that publish or change a lot of content daily.
- Sites with many “Discovered – currently not indexed” URLs in Search Console.
What wastes crawl budget#
- Parameter URLs from filters, sorting and session IDs.
- Internal search result pages.
- Redirect chains and broken links.
- Soft 404s and thin, near-duplicate pages.
- Slow server responses and frequent 5xx errors.
How to optimise it#
- Block crawl traps with robots.txt (e.g. faceted parameters, internal search).
- Consolidate duplicates with canonical tags and redirects.
- Improve server speed — faster responses let Google crawl more.
- Keep the sitemap clean and internal links pointing at final URLs.
- Remove or improve low-value pages so demand concentrates on what matters.
Monitor in Search Console → Settings → Crawl stats. Related: indexing.