Crawl Budget
Crawl budget is how many pages a crawler fetches from your site in a period. Learn what sets it, when it matters, and how to stop wasting it.
TL;DR
Crawl budget is the number of URLs a crawler will fetch from your site in a given period. It emerges from two things: how much load your server can absorb without slowing down, and how much the engine wants your content. For most sites it is not a constraint worth thinking about. For large or parameter-heavy sites it decides how quickly anything you publish gets seen.
Why crawl budget matters
When a crawler has a finite allocation and your site offers far more URLs than it will ever fetch, what it spends that allocation on becomes your problem.
- Indexing speed: New and updated pages wait longer to be discovered when crawlers are busy with low-value URLs.
- Freshness accuracy: Engines repeat what they last fetched, so slow re-crawling means stale facts persist in answers.
- Large site coverage: Beyond a few hundred thousand URLs, sections can go uncrawled for months.
- Server cost: AI crawlers have added meaningful load, making efficiency a hosting concern as well as a visibility one.
How to manage crawl budget
Find the waste first
Analyse server logs by user agent. The URLs crawlers actually fetch are rarely the ones teams assume.
Block infinite spaces
Internal search results, calendars and filter combinations generate unlimited URLs and deserve a robots.txt disallow.
Fix redirect chains
Every hop costs a fetch. Point redirects straight at the final destination rather than through intermediate URLs.
Remove dead URLs from discovery
404s and redirects sitting in sitemaps and internal links consume budget for nothing.
Improve response times
Faster servers raise the crawl rate limit, since engines throttle to avoid degrading your site.
Consolidate duplicates
Canonicals and parameter handling stop crawlers fetching several versions of one page.
Crawl budget vs. index coverage
Crawl budget: How many URLs get fetched. A question about crawler attention and server capacity.
Index coverage: How many fetched URLs get included in the index. A question about page quality and duplication.
They fail differently. A page that is never crawled has a discovery problem; a page crawled and excluded has a quality problem. Reading them as one number leads to fixing the wrong thing.
Log analysis by user agent is where crawl problems become visible, and few teams do it. Findrix checks crawler access and technical health across 31 points, then tracks whether seven AI engines cite your pages against named competitors. Every gap comes with the fix already written: technical, content and off-site. The audit is free, takes about a minute, and requires no signup.
What wastes crawl budget
- Faceted navigation: Filter combinations producing thousands of near-identical URLs.
- Session and tracking parameters: The same page at endless distinct addresses.
- Redirect chains: Multiple hops where one would do.
- Soft 404s: Empty pages returning 200 status codes, which crawlers keep revisiting.
- Duplicate content: Several URLs serving the same thing without canonical consolidation.
Whether this applies to you at all
Google has said plainly that most sites do not need to think about crawl budget. Under a few thousand URLs with reasonable server performance, everything worth crawling gets crawled and the topic is a distraction.
It becomes real above roughly ten thousand URLs, or wherever faceted navigation and parameters multiply your URL space beyond your actual page count, or when publishing frequency outpaces crawl frequency.
AI crawlers add load without the traffic return that made Googlebot feel worthwhile, so sites that never considered crawl efficiency are now looking at bandwidth bills.
Frequently asked questions
Does my site need crawl budget optimization?
Probably not if you have fewer than a few thousand URLs and a reasonably fast server. It matters for large sites, ecommerce catalogues with faceted navigation, and anything generating URLs from parameters.
Do AI crawlers have a crawl budget?
They have their own rates and priorities, which are not published the way Google documents its behaviour. The practical implication is the same: fetches are finite, so making sure they land on pages worth citing is the useful work.
How do I see my crawl budget?
Search Console Crawl Stats shows Googlebot activity. Server log analysis is more complete and covers every crawler including AI bots, which is the only way to see what they are actually fetching.