{"id":6397,"date":"2026-06-01T16:43:57","date_gmt":"2026-06-01T16:43:57","guid":{"rendered":"https:\/\/www.wpconsults.com\/?p=6397"},"modified":"2026-06-01T16:44:48","modified_gmt":"2026-06-01T16:44:48","slug":"analyse-des-fichiers-journaux-log-file-des-robots-dindexation-crawlers","status":"publish","type":"post","link":"https:\/\/www.wpconsults.com\/fr\/log-file-analysis-ai-crawlers\/","title":{"rendered":"Les journaux de serveur sont les seuls enregistrements honn\u00eates des robots d'indexation de l'IA : un manuel d'audit de r\u00e9f\u00e9rence crois\u00e9e"},"content":{"rendered":"\n

Here is the uncomfortable truth most “AI visibility” dashboards will not tell you: the analytics product you trust cannot see the crawlers you are trying to measure. <\/p>\n\n\n

\"GA4<\/figure>\n\n\n

Google Analytics 4 fires on JavaScript execution. AI crawlers, almost without exception, do not execute JavaScript. They request the raw HTML, take what they want, and leave. Which means the single source of truth for what GPTBot, ClaudeBot, PerplexityBot and the rest actually did on your site is sitting in a file your developer probably rotates and deletes every 14 days: the server access log.<\/p>\n\n\n\n

Log file analysis for AI crawlers is the only honest record of who fetched what, when, and whether they were even real.<\/strong> <\/mark>Everything else is inference. This is the audit playbook I run when a client wants to know what the answer engines are doing to their site, and why most of the “our content is being trained on” panic is either unverified or flat wrong.<\/p>\n\n\n\n

Why GA4 and most analytics are structurally blind to AI bots?<\/h2>\n\n\n\n

GA4 is a client-side measurement tool. It loads a tag, the tag runs in a browser-like environment, and an event is sent. No JavaScript execution, no event. AI crawlers behave like classic HTTP clients: they issue a GET, parse the markup server-side, and never touch your tag. So your behavioural analytics will show zero sessions from GPTBot even while your logs show tens of thousands of its requests. People then conclude “AI is not crawling us.” It is. You are just measuring with the wrong instrument.<\/p>\n\n\n\n

This is the same architectural reason your critical content has to live in the raw HTML, not in a client-rendered shell. If a passage only appears after JavaScript hydration, an AI crawler that does not execute JS will never ingest it. The logs make this visible in a way no rendering test does: you see exactly which URLs the bot hit and, by cross-referencing the response your server returned, exactly what bytes it received.<\/p>\n\n\n\n

Training crawlers vs live retrieval agents: stop treating them as one thing<\/h2>\n\n\n\n

The biggest analytical error I see is lumping every AI user-agent into one “AI bot” bucket. They serve completely different functions and demand different decisions from you. Broadly there are two classes.<\/p>\n\n\n\n

Training crawlers<\/strong> harvest content to build or refine foundation models. They are bulk, systematic, and indifferent to whether a human is waiting. This group includes GPTBot<\/strong> (OpenAI), ClaudeBot<\/strong> (Anthropic), CCBot<\/strong> (Common Crawl, which many models ingest downstream), and the access controlled by Google-Extended<\/strong> (Google’s token for Gemini training, which is a robots.txt directive rather than a separate crawling user-agent). Blocking these affects whether your content feeds the next model. It does not affect whether you appear in a live answer today.<\/p>\n\n\n\n

Live retrieval agents<\/strong> fetch a page because a user just asked a question and the engine needs a citation right now. This is the group that actually drives AI referral visibility: ChatGPT-User<\/strong> (OpenAI’s on-demand fetch when a user prompts ChatGPT to browse), OAI-SearchBot<\/strong> (OpenAI’s index for ChatGPT search results), and PerplexityBot<\/strong> (Perplexity’s retrieval). If you block these, you remove yourself from the answer. Many sites blanket-block “AI” in robots.txt, kill OAI-SearchBot and ChatGPT-User along with GPTBot, and then wonder why they vanished from ChatGPT citations. They shot the messenger and the customer.<\/p>\n\n\n\n

OpenAI itself documents this separation and the independent control it gives you: you can allow OAI-SearchBot to appear in search while disallowing GPTBot to opt out of training.[1]<\/sup> Treat the two classes as one and every downstream decision you make is wrong.<\/p>\n\n\n\n

A working user-agent cheat sheet<\/h3>\n\n\n\n
User-agent token<\/th>Operator<\/th>Class<\/th>What blocking it costs you<\/th><\/tr><\/thead>
GPTBot<\/td>OpenAI<\/td>Training<\/td>Out of future model training data only<\/td><\/tr>
OAI-SearchBot<\/td>OpenAI<\/td>Retrieval \/ index<\/td>Out of ChatGPT search results<\/td><\/tr>
ChatGPT-User<\/td>OpenAI<\/td>Live retrieval<\/td>Cannot be fetched when a user asks ChatGPT to browse<\/td><\/tr>
ClaudeBot<\/td>Anthropic<\/td>Training<\/td>Out of future Claude training data<\/td><\/tr>
PerplexityBot<\/td>Perplexity<\/td>Retrieval \/ index<\/td>Out of Perplexity answers and citations<\/td><\/tr>
CCBot<\/td>Common Crawl<\/td>Training (upstream)<\/td>Out of a dataset many models ingest<\/td><\/tr>
Google-Extended<\/td>Google<\/td>Training control (robots token)<\/td>Out of Gemini training; does not affect Search<\/td><\/tr><\/tbody><\/table>
Separate the bots by function before you touch robots.txt. Blocking a retrieval agent is not the same decision as blocking a training crawler.<\/figcaption><\/figure>\n\n\n\n

The growth that makes this non-optional<\/h2>\n\n\n\n

This was a footnote two years ago. It is now a budget line. Cloudflare’s network-wide analysis found that between May 2024 and May 2025, GPTBot requests grew roughly 305%<\/strong> while overall Googlebot requests grew about 96%. The more striking number is the live-retrieval side: ChatGPT-User requests surged roughly 2,825%<\/strong> over the same period, reflecting how often users now ask ChatGPT to go fetch a live page.[2]<\/sup><\/p>\n\n\n\n

\"Bar
AI crawler request growth, May 2024 to May 2025. Source: Cloudflare Radar.<\/figcaption><\/figure>\n\n\n\n

A near 30x increase in one retrieval agent is not noise you can ignore on a shared host. It is real bandwidth, real origin load, and real crawl-budget competition. Which brings us to the second hard truth: a large share of traffic claiming to be these bots is lying.<\/p>\n\n\n\n

Reverse-DNS verification: most “AI bot” traffic in your logs is spoofed<\/h2>\n\n\n\n

A user-agent string is a request header. Anyone can set it. Setting User-Agent: GPTBot<\/code> is a one-line change, and scrapers, paywall-jumpers and competitors do it constantly because the entire allow-by-user-agent model naively trusts the claim. If you build a crawl report straight off the user-agent field, you are reporting fiction. Verification is not optional; it is the first filtering step before any number you produce means anything.<\/p>\n\n\n\n

There are two reliable methods, in order of preference.<\/p>\n\n\n\n

1. Published IP range files.<\/strong> The serious operators publish machine-readable IP lists you can match against. OpenAI publishes gptbot.json<\/code>, searchbot.json<\/code> and chatgpt-user.json<\/code>; Common Crawl publishes its ranges; Google publishes its crawler IP lists. Match the request’s source IP against the relevant file. If it is not in the list, the user-agent is lying, full stop. This is the cleanest check because it does not depend on DNS at all.<\/p>\n\n\n\n

2. Reverse-DNS plus forward-confirm.<\/strong> For vendors that do not publish IP files (Anthropic’s ClaudeBot is the notable case), use the same forward-confirmed reverse DNS technique Google has recommended for verifying Googlebot for years.[3]<\/sup> <\/p>\n\n\n\n

The logic: do a reverse lookup on the source IP to get a hostname, confirm the hostname belongs to the claimed operator, then do a forward lookup on that hostname and confirm it resolves back to the original IP. Both directions must agree.<\/p>\n\n\n\n

# Step 1: reverse lookup the IP that claimed to be a bot\ndig -x 66.249.66.1 +short\n# -> crawl-66-249-66-1.googlebot.com.\n\n# Step 2: forward lookup that hostname\ndig crawl-66-249-66-1.googlebot.com +short\n# -> 66.249.66.1   (matches: verified)\n\n# If the hostname does not belong to the operator,\n# or the forward lookup does not return the original IP,\n# the request is spoofed. Discard it before reporting.<\/code><\/pre>\n\n\n\n
Run this against a sample of every user-agent you care about. On most sites I audit, a meaningful slice of “GPTBot” and “PerplexityBot” hits fail verification. Reporting unverified user-agents as real AI crawl activity is the single most common way these audits mislead the people paying for them.<\/p>\n\n\n\n
The cross-reference: logs vs crawl vs analytics<\/h2>\n\n\n\n
A single data source lies by omission. The method that actually produces decisions is a three-way reconciliation. Each source answers a different question, and the gaps between them are where the insight lives.<\/p>\n\n\n\n
    \n
  1. Server logs<\/strong> answer: what did bots and users actually request, and what status code did we return? This is ground truth for behaviour.<\/li>\n\n\n\n
  2. A crawler’s own crawl<\/strong> (Screaming Frog, Sitebulb, or your sitemap export) answers: what URLs do we believe exist and should be reachable?<\/li>\n\n\n\n
  3. Analytics and Search Console<\/strong> answer: what did humans engage with, and what drove value?<\/li>\n<\/ol>\n\n\n\n
    Lay the three side by side, keyed on URL, and read the diffs:<\/p>\n\n\n\n
    In logs?<\/th>In crawl\/sitemap?<\/th>In analytics?<\/th>What it means<\/th><\/tr><\/thead>
    Yes (AI bot)<\/td>Yes<\/td>No human traffic<\/td>AI ingests it but humans do not land. Candidate for AI-only value, or thin content the bot wastes budget on.<\/td><\/tr>
    Yes (AI bot, heavy)<\/td>No<\/td>No<\/td>Bot is hammering URLs you do not even list: parameter explosions, faceted filters, old paginated junk. Crawl-budget waste.<\/td><\/tr>
    No<\/td>Yes<\/td>Yes<\/td>Important page no AI crawler has fetched. Check robots.txt, internal links, and that it lives in raw HTML.<\/td><\/tr>
    Yes (returns 404\/5xx)<\/td>Yes<\/td>n\/a<\/td>You are feeding errors to AI crawlers. They learn your site is broken; retrieval agents drop you from answers.<\/td><\/tr><\/tbody><\/table>
    The decisions live in the disagreements between the three sources, not in any single one.<\/figcaption><\/figure>\n\n\n\n
    A concrete, repeatable methodology: export your access log for a clean 30-day window, filter to verified bot hits only, normalise the URL (strip session params you do not want counted), then left-join your sitemap and your Search Console export on the URL key. Group by user-agent class (training vs retrieval) and by status code. <\/p>\n\n\n\n
    In an afternoon you will know which URLs the answer engines actually fetch, which ones they waste requests on, and which of your money pages they have never touched. That is a far cry from “AI is crawling us a lot.”<\/p>\n\n\n\n
    Spotting crawl-budget waste before it costs you<\/h2>\n\n\n\n
    Crawl budget was a Googlebot conversation. It is now an AI-bot conversation too, and the AI crawlers are far less disciplined. The waste signatures to hunt in the verified-bot logs:<\/p>\n\n\n\n