Google rewrote its crawl budget guide on July 22, 2026, and almost every page currently explaining crawl budget for eCommerce sites now predates it. The one line that matters most for a store is new: your crawl capacity is a single pool shared across all of Google’s crawlers, so your feed, image, and ads crawling compete with product discovery.
Here is my honest read after reading the new doc in full and testing it on our own site. Most stores do not have a crawl budget problem at all, and the ones that do usually have a duplicate URL problem wearing a crawl budget costume.
Key Takeaways
- Google rewrote its crawl budget doc on July 22, 2026 and now states that one crawl capacity limit is shared across all of its crawlers.
- Google Shopping crawls harder when you have merchant feeds and AdsBot crawls harder with dynamic ads, so both draw from the same pool as product discovery.
- Most stores do not have a crawl budget problem; Google says if your pages get crawled the same day they publish, you can skip the topic.
- When a store does have one, the usual cause is duplicate URL generation (tags, filters, sort parameters, pagination), not a low ceiling.
- Supporting HTTP 304 saves crawl, but whether you “support 304” depends on which validator your server sends; our origin honors it by ETag and not by If-Modified-Since.
- You cannot request more crawl budget; Google raises the limit only when your server keeps responding fast and healthy.
What crawl budget actually is for an online store
Crawl budget is the set of URLs Google can and wants to crawl on your site. It is set by two things: your crawl capacity limit (how much crawling your server can take) and Google’s crawl demand (how much Google actually wants to crawl your pages).
Think of the capacity limit like a busy shop door. It is how many people Google will let in at once, plus how long it waits between visits, set so it does not overwhelm your server. That number goes up only if your server keeps answering quickly, and it drops if your server starts to struggle.
Crawl demand is the other half: how badly Google wants your pages, driven by how popular they are and how often they change. A store adding and editing products daily creates real demand; a static brochure site creates very little. If you want the measurement side of this, our older guide on finding crawl budget and its use cases walks through reading the numbers in Search Console.
Does your store even have a crawl budget problem?
Most stores do not, and Google says so on the same page everyone uses to sell crawl budget optimization. The rewritten doc states plainly that if your pages “seem to be crawled the same day that they are published, you don’t need to read this guide.”
Google frames the whole guide as advanced material for three kinds of site: large sites with a million or more URLs that change weekly, medium or larger sites with 10,000 or more URLs that change daily, and sites where a big share of URLs sit in Search Console as Discovered, currently not indexed. It also adds that these numbers are “a rough estimate” and “not exact thresholds”, so treat them as a smell test, not a rule.
| You can ignore crawl budget if | You should pay attention if |
|---|---|
| New products get crawled the same day you publish them | You run a large catalog (roughly 1 million or more URLs) that changes weekly |
| Your store is small to mid sized (under about 10,000 URLs) | You have 10,000 or more URLs and content changes daily |
| Few URLs sit in “Discovered, currently not indexed” | A large share of your URLs sit in “Discovered, currently not indexed” |
What Google’s rewritten crawl budget doc changed in July 2026
Most of the July 22 rewrite was housekeeping. Google described it as improving “clarity, terminology consistency, and flow” and making sure a term like crawl capacity limit is used the same way throughout, which Search Engine Roundtable flagged the morning it landed.
But a few lines are genuinely new, and one of them is the reason this piece exists. The doc also quietly moved: it used to live under Search documentation and now sits in a broader crawling section, which fits its own point that the same budget is spent by Search, Shopping, and Ads crawlers.
| New or sharpened line | Why it matters for a store |
|---|---|
| One crawl capacity limit is shared across all crawlers | Your feed, image, and ads crawling compete with product discovery |
| Google Shopping crawls harder with merchant feeds; AdsBot crawls harder with dynamic ads | Names the exact store crawlers drawing on the shared pool |
| Every site starts on the same default, conservative capacity limit | Google raises it only as your server proves it can take more |
| Support HTTP 304 (Not Modified) to avoid re-downloading unchanged pages | Testable on your own origin, and most stores never return a 304 |
Why one crawl capacity limit is shared across all of Google’s crawlers
Google’s rewritten doc now says it directly: “the crawl capacity limit is shared across all crawlers”, and “high demand from one crawler can reduce the capacity available for others.” That is one sentence, and it changes how you should think about crawling a store.
The doc then names the crawlers that matter to a shop. It says AdsBot “generally has a higher demand when a site is running dynamic ad targets, and Google Shopping has a higher demand for products you have in your merchant feeds.” Read those two lines together and the picture is clear: Googlebot for page discovery, Googlebot-Image for your product photos, Google Shopping for your feed, and AdsBot for your ad landing pages all pull from the same host budget.
This is Google describing its own systems, which is the strongest kind of vendor statement and still a vendor statement, so I take it as its account of how crawling works rather than a settled law. What we can show first-hand is the split on our own property. Even with no Merchant Center feed running, our Search Console Crawl Stats break the crawl out by crawler type, and AdsBot alone accounts for 9% of it, with image and page-resource fetching taking a big slice more.
On a store, that mix skews harder toward feed and image crawling than a content site like ours does. If you want to see how one of those store-specific crawlers behaves in the wild, we documented that separately in our look at Google StoreBot.
How a store can starve its own product discovery
A store with a large image catalog, an active merchant feed, and Shopping or Performance Max ads can have its product pages discovered slowly, and not because of any on-page SEO mistake. It happens because its own feed, image, and ads crawling is spending the shared pool before Googlebot gets to the new products.
Our own data hints at why this bites. In Crawl Stats, 90% of Google’s crawl on our site is refreshing pages it already knows, and only 10% is discovery, the slice that finds brand new URLs. On a store publishing products every day, that thin discovery slice is exactly what your feed and image crawling are competing with.
How a store spends its shared crawl budget
- Googlebot, Googlebot-Image, Google Shopping, and AdsBot all draw on one shared capacity pool
- Heavy image, feed, and ads crawling takes a large share of that pool
- Less capacity is left for Googlebot to discover and refresh pages
- New and updated product pages get found and refreshed more slowly
Does supporting HTTP 304 actually save crawl budget?
Yes, when it genuinely works. Google’s doc adds a new line asking sites to “support 304 (Not Modified) HTTP status codes”, because a 304 tells Google to reuse the copy it already has instead of downloading the page again. But “supporting 304” is not one switch you flip; it depends on which validator your server sends.
A 304 works through conditional requests. Google remembers a page’s ETag (a content fingerprint) or its Last-Modified date, then asks “has this changed since?” If nothing changed, the server should answer 304 and send no body. We tested exactly that on our own origin, which serves from a LiteSpeed page cache, and the result was more mixed than the one-line instruction suggests.
| Conditional request we sent | What our origin returned |
|---|---|
| If-None-Match with a blog post’s own ETag | HTTP 304 Not Modified |
| If-None-Match with the homepage ETag | HTTP 304 Not Modified |
| If-Modified-Since with a recent date | HTTP 200, the full page again |
So we “support 304” by ETag and not by If-Modified-Since, purely because our pages ship an ETag header and no Last-Modified one. There is nothing for an If-Modified-Since request to check against, so the server hands back the whole page. Our Crawl Stats agree with that: Not modified (304) is under 1% of all crawl responses on our site, which means Google is re-downloading almost everything even when it has not changed.
The takeaway for a store is to test it yourself rather than assume. Send the two conditional requests to a product page and see which your platform honors, because an image-heavy or feed-heavy store that never returns a 304 is spending capacity re-fetching bytes that did not change. This is the same headers-level check we used in our guide on how to check if your CDN is caching.
The real problem for most stores is duplicate URL generation
For most stores that genuinely crawl slowly, the disease is not a low ceiling, it is too many low-value URLs. Google’s doc calls this “perceived inventory” and says it is “the factor that you can positively control the most”, which is a polite way of saying you are usually the one generating the waste.
On a store, those URLs appear without anyone creating them. Every tag on a collection gets its own URL, every storefront filter adds one, every sort order adds another, and each of those paginates. Google then spends its crawl on near-identical pages instead of your real products. We pulled that mechanism apart in detail for one platform in our piece on Shopify duplicate content from collections, and the shape is the same on WooCommerce.
Seen this way, crawl budget is the symptom and duplicate URL generation is the illness. Fix the URL explosion and the “budget problem” usually dissolves on its own, which is why I would look there long before I worried about the capacity limit.
How to manage crawl budget on an online store
The levers Google’s doc actually hands you are narrow: cut duplicate and junk URLs, keep the server fast and healthy, and point crawling at what matters. There is no button to request more budget, so ignore anyone who implies there is.
- Consolidate duplicate content so Google spends crawl “on unique content rather than unique URLs”, in its words. Canonicals, tidy faceted navigation, and not indexing filter and sort URLs do most of the work here.
- Return a 404 or 410 for products you have permanently removed, because Google calls a 404 “a strong signal not to crawl that URL again”, so it stops wasting fetches on dead SKUs.
- Avoid long redirect chains, which Google says “have a negative effect on crawling”; a migrated store full of hop-through redirects burns crawl on every one. Our guide on finding and flattening redirect chains covers the cleanup.
- Keep the server fast and stable, because crawl health is the one input that raises your limit; steady response times and time-to-first-byte push it up, while 5xx errors and 429 rate-limit responses push it down.
- Do not lean on robots.txt to “free up budget”. Google is explicit that it “won’t shift this newly available crawl budget to other pages unless Google is already hitting your site’s crawl capacity limit”, so blocking junk URLs helps focus, not headroom.
- Do not use noindex to save crawl either; Google still requests the page, then drops it once it sees the tag, so you have already spent the fetch.
Watch all of this where the real numbers live, in Search Console Crawl Stats and your server logs. On our own site the host status reads healthy, with a 460 ms average response time and no host problems, which is the state that lets the limit rise rather than fall.
If you want to go deeper than the summary, your raw server logs show exactly which crawler hit which URL and when. We covered reading them, including telling the crawler types apart, in our guide to log file analysis for AI and search crawlers.
So, should your store worry about crawl budget?
Honestly, for most stores the answer is no, and the rewritten doc quietly agrees. If your products get crawled the day you publish them and Search Console is not full of “Discovered, currently not indexed” URLs, your time is far better spent on content and links than on chasing a crawl ceiling you are not hitting.
The stores that should care are the big, fast-moving catalogs, and for them the real July 2026 news is the shared pool: your feed, image, and ads crawling now visibly competes with product discovery, so a bloated URL set or a heavy feed is not a separate issue from “Google is slow to find my products”, it is the cause. My advice is boring on purpose. Check whether you even have the problem first, then fix duplicate URLs and server health before you touch anything clever, because that is where the actual budget goes.
Common questions about crawl budget for stores
How do I check my store’s crawl budget?
Open Search Console, go to Settings, then Crawl stats. It shows your total crawl requests, average response time, host status, and a breakdown by Googlebot type, which is the closest thing Google gives you to a live view of how your budget is being spent.
Will blocking filter URLs in robots.txt free up crawl budget for my products?
Not by itself. Google says it will not move freed-up crawl to other pages unless you are already hitting your capacity limit. Blocking junk URLs helps Google focus on the right pages, but it does not hand you extra headroom.
What is the crawl capacity limit?
It is the maximum amount of simultaneous crawling Google will do on your site, plus the delay it leaves between fetches, set so it does not overload your server. Every site starts on the same conservative default, and Google raises it only as your server proves it can handle more.
Work with WpConsults
Not sure whether your store has a crawl budget problem?
If your product pages are slow to get found, or a big chunk sits in “Discovered, currently not indexed”, I can read your Crawl Stats and logs and tell you whether it is really crawl budget or a duplicate URL problem in disguise.
Update Logs
23 Jul 2026
- First published, built on Google’s July 22, 2026 crawl budget rewrite, with our own Crawl Stats and a first-hand HTTP 304 test on our origin.
Want our posts to show up more often on Google?
One step & Google will surface this site in your Top Stories.
