Google StoreBot: What It Crawls, Why It Gets Blocked, and How to Fix Access Issues

Google StoreBot is the crawler that quietly walks your product pages, cart, and checkout to confirm your live prices and availability match what you sent to Google Merchant Center. When it gets blocked, your Shopping listings start throwing warnings, and most guides online stop at a one-line definition.

This guide covers what StoreBot actually does, how to spot Storebot-Google in your logs, the exact ways it gets blocked, and how to unblock it without hurting your listings. Google updated its fix documentation this month, and I have read the current version so you do not have to.

What Google StoreBot is and what it crawls

Google StoreBot is Google’s specialized shopping crawler. It goes through your product pages, adds items to the cart, and walks the full checkout flow without ever buying, so it can check that the price, availability, and shipping on your live site match what you submitted to Google Merchant Center.

Google describes it plainly in its own documentation. The crawler uses machine learning to fill in forms like shipping addresses, then catalogs details such as product price, shipping price, online and local availability, coupon validity, shipping time, payment methods, and whether your address and payment steps are secure.

So the job is verification, not ranking. This is the part most store owners get wrong, so it is worth stating clearly.

StoreBot feeds Google Shopping. Google’s crawler documentation says crawling preferences for the Storebot-Google user agent affect all surfaces of Google Shopping, like the Shopping tab in Search and Google Shopping itself.

Nothing in Google’s documentation connects StoreBot to your organic web ranking. If a blog post drops in Search, StoreBot is not why. It cares about your feed and your checkout, which is a different system from the one that ranks your articles.

One useful detail sits at the end of Google’s description: structured data takes precedence over what the crawler extracts. If you add accurate product schema to your pages and correct data in Merchant Center, that overrides whatever StoreBot pulls off the page.

How to identify Storebot-Google in your server logs

You spot StoreBot two ways: by its user agent string in your server logs, and by a fake shopper moving through checkout without buying. The robots.txt token and log identifier is Storebot-Google.

Google publishes two user agent strings, one for desktop and one for mobile. Here they are exactly as Google lists them.

Notice the Chrome/W.X.Y.Z part. That W.X.Y.Z is a placeholder for a version number that keeps changing, so in your log filters you wildcard it rather than pin a version. This matters more than it sounds, and I will come back to it.

If you would rather not read logs, StoreBot also shows up inside your store as a shopper who never converts. Google says you may see a user with the first name Google and the last name StoreBot advancing through your checkout without making a purchase. That is the crawler, self-identifying.

Here is the trap: the user agent is a single line of text, and anyone can forge it. A hit that says Storebot-Google is not proof that Google sent it, so before you trust it or block it, verify.

Google itself recommends a reverse DNS lookup over static IP allowlisting. StoreBot crawls from Google’s published IP ranges, and a genuine hit resolves to a crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com host.

I wrote a full verify-then-count method for AI and shopping crawlers in our log file analysis guide, and it applies here directly. Filter your logs for Storebot-Google, then confirm each hit against Google’s ranges before you believe a single number.

This is exactly why the wildcard matters. Some bot directories still print StoreBot’s old hardcoded Chrome/79 user agent from years ago. Build a log filter or a firewall rule from that stale string, and it silently misses today’s real StoreBot requests, so your “StoreBot is blocked” or “StoreBot never visited” conclusion is wrong before you start.

Why Google StoreBot gets blocked, and how to fix access

Most StoreBot access problems come down to five things, and Google names all five in its updated fix documentation: robots.txt rules, user-agent blocklists, IP or firewall filtering, bot fingerprinting, and slow pages that time out the crawler.

Here is what each of the five block modes actually does, so you know where to look:

• robots.txt: your file disallows the crawler, either by name or by a catch-all rule, so StoreBot is told to stay out before it fetches a thing.
• User-agent blocklists: out-of-the-box bot detection ships with a list of crawler user agents to block, and StoreBot can be on it by default.
• IP and firewall rules: a firewall or a CDN like Cloudflare returns 403 or 404 to automated clients, so StoreBot hits a wall a human never sees.
• Fingerprinting: bot detection builds a signature from the user agent, IP, browser JavaScript variables, and crawl rate, then blocks what looks automated, which is exactly what a form-filling checkout crawl looks like.
• Slow pages: slow landing pages cause crawler timeouts, so a page that loads fine for you can read as inaccessible to StoreBot.

The robots.txt gap most StoreBot fixes miss

This is the one that catches careful store owners, because you can follow Google’s own fix doc to the letter and still stay blocked.

Google’s fix document tells you to make sure robots.txt allows Google, and its example says to add User-agent: Googlebot and User-agent: Googlebot-image with an empty Disallow. It never mentions Storebot-Google, which is StoreBot’s actual robots.txt token per Google’s own crawler list.

So picture a store with a strict robots.txt: it allows Googlebot, then blocks everything else with User-agent: * and Disallow: /. That file follows the fix doc, passes a quick read, and still blocks StoreBot, because the catch-all disallow catches Storebot-Google along with everything else.

I am not saying Google’s advice is wrong; I am saying the fix doc is incomplete on its own robots.txt example. The complete version names StoreBot’s token explicitly, alongside Googlebot:

User-agent: Googlebot
Disallow:

User-agent: Googlebot-image
Disallow:

User-agent: Storebot-Google
Disallow:

If your robots.txt already allows all reputable crawlers by default, you are fine. The problem only bites stores that lock robots.txt down to a named allowlist and forget StoreBot has its own name.

Bot detection, blocklists, and fingerprinting

Bot mitigation software is the second and fourth block modes, and on a modern store it is the cause I would check first, because so many shops now run some kind of protection.

Out-of-the-box bot detection ships with blocklists of crawler user agents, and StoreBot can land on one by default. Fingerprinting goes further: it builds a signature from the user agent, IP, browser JavaScript variables, and the rate a given IP is crawling, then blocks what reads as a bot.

StoreBot filling forms and moving through checkout at machine speed is exactly the pattern these tools flag. If you run Cloudflare, a WAF, or a plugin-based firewall, that is where to look first.

Check your bot-management and challenge rules for anything catching the checkout simulation, then allow verified Google crawlers rather than fighting them. On WooCommerce this usually means a Cloudflare or security-plugin rule; on Shopify the platform handles most of it, though a strict third-party bot app can still interfere.

Firewall, CDN, and IP filtering

Third block mode. If you sit behind a firewall or a CDN, your server can quietly return 403 or 404 only to automated clients, so a human sees a working checkout while StoreBot sees a wall.

The fix Google recommends runs against a habit a lot of stores have. Do not statically allowlist Google’s search crawler IP addresses, because StoreBot can use different IPs than the ones you allowlisted.

Use a reverse DNS lookup to verify the IP belongs to Google instead. That is the same verification step from the logs section, and it is the durable one, because Google’s IP ranges change but the reverse DNS pattern does not.

Slow pages that time out the crawler

The fifth mode is the easiest to overlook because it does not feel like a block. Slow landing pages cause crawler timeouts, so a page that loads fine for you but crawls slowly can read as inaccessible to StoreBot.

Google points to PageSpeed Insights to find and fix the culprits. On a big catalog this is worth a real look, because a slow product template multiplied across thousands of SKUs is a lot of timed-out crawls.

How to trigger a StoreBot recrawl in Merchant Center

Once access is restored, you do not just sit and wait. Trigger a recrawl in Merchant Center: go to Diagnostics and select Request website check or Request review for a bulk recrawl, or use Fetch now in the Feeds section for a feed.

Then give it time. Google says it typically takes 12 to 48 hours for the system to reflect the changes and re-approve your products.

If you are still warned after two days, the access issue probably is not actually fixed. Go back to your logs and confirm a verified StoreBot hit is getting a 200, not a 403, before you assume the recrawl failed.

The fake shopper in your checkout: StoreBot and abandoned carts

Long before StoreBot identified itself, store owners kept finding the same ghost in their orders: a shopper who filled the cart, started checkout, and vanished. As reported by the Wall Street Journal (via Lunio’s retelling), a mystery shopper named John Smith leaving abandoned carts on Shopify stores was confirmed to be Google.

The practical damage store owners have reported is real even if the name has changed. A crawler walking checkout inflates your abandoned-cart count, and any automated recovery emails fire at a fake address that bounces, which over time can hurt your sending reputation.

The good news is the ghost now wears a name tag. Google’s current documentation says the checkout shopper shows up with the first name Google and the last name StoreBot, so you can filter it out of your abandoned-cart reports and stop chasing it with recovery emails.

If your cart analytics let you exclude by name or by the Storebot-Google user agent, do it, so your real abandonment rate is not skewed by a crawler doing its job.

Should you block Google StoreBot?

For almost every store, no. Blocking StoreBot is how you get Merchant Center warnings and, if it drags on, suspended free listings and local inventory ads, which costs you far more than the crawler ever does.

There is one honest wrinkle worth knowing, because the sources disagree. Google’s crawler documentation says its common crawlers, StoreBot included, always obey robots.txt when crawling automatically. DataDome, a bot-mitigation vendor that watches this traffic across its own network, logs StoreBot’s robots.txt compliance as not respected.

Both are interested parties: Google is describing its own crawler, and DataDome sells bot mitigation. I have not tested this myself, so I will not hand you a verdict I cannot stand behind.

One reconciliation sits in Google’s own wording. It says the crawler obeys robots.txt “when crawling automatically”, and a form-filling checkout simulation may not be plain automatic crawling. That is a scoped phrase, not a blanket promise.

Here is the move that is true under either reading: do not rely on robots.txt alone to control StoreBot. If you want it out of a specific area, add the rule and then confirm in your verified logs that it actually stopped fetching, rather than assuming the rule worked.

And if you are trying to keep your listings healthy, the opposite is the point: let it crawl, and make sure nothing in your stack is quietly blocking it. Our Merchant Center requirements guide covers the rest of what keeps your feed approved.

So, should you let Google StoreBot crawl your store?

In my view, yes. For almost every store this is a crawler you want, not one you fight. It is the mechanism that keeps your Shopping data honest, and the cost of blocking it, warnings and suspended listings, is far worse than a slightly inflated abandoned-cart number.

The one thing I would not do is trust a single log line or a single vendor claim about it. Verify StoreBot hits against Google’s own ranges before you act, treat the “always obeys robots.txt” line as Google’s position rather than settled fact, and if you must restrict it, confirm the restriction in your logs instead of assuming.

Get that right, and StoreBot goes back to being invisible, which is exactly where a working crawler belongs.

---

Share Copy link Copy as Markdown

Want our posts to show up more often on Google?

One step & Google will surface this site in your Top Stories.

Add us as a preferred source on Google

Abdullah Nouman eCommerce SEO and AI Search Marketing Specialist

Abdullah Nouman is an eCommerce SEO and AI Search Marketing specialist and the founder of WpConsults. He started as a web developer in 2017 then shifted to search marketing, so he does not just advise, he implements. He helps Shopify and WooCommerce stores rank on Google and get cited by ChatGPT with a focus on the shift toward AI agentic shopping.

Work with him

Related reading

Back to all posts →

How to Check If Your CDN Is Actually Caching Your Pages WP-Cron Alternatives Compared: Server Cron vs Action Scheduler vs External Services Shopify Duplicate Content From Collections: The Tag, Filter, and Pagination URLs You Never Created