# Daily snapshot 2026-11-18

The numbers come from one website. They describe what these crawlers did here, not across the web.

Period: 2026-11-18T00:00:00Z to 2026-11-19T00:00:00Z (00:00Z to 24:00Z). Generated 2026-11-19T03:18:24.094Z. Schema version 1. Licence CC BY 4.0. Cite as: Benes, Peter. Rattlesnakes By Mail: A Field Guide to the Crawlers of the Web. benesthemenace.com. Method: https://rattlesnakesbymail.com/method

Requests on 2026-11-18 by crawler, against each crawler's 28-day median. Googlebot: 58 requests, 28-day median 44; Bingbot: 31 requests, 28-day median 38; GPTBot: 6 requests, 28-day median 4; OAI-SearchBot: 4 requests, 28-day median 6; PerplexityBot: 3 requests, 28-day median 0; GoogleOther: 2 requests, 28-day median 0; Reflectionbot: 2 requests, 28-day median 0.

## Facts

- F-261118-01 Bingbot first fetched the 2026-11-17 snapshot 4 h 53 min after it was published. https://rattlesnakesbymail.com/observed/bingbot
- F-261118-02 Googlebot first fetched the 2026-11-17 snapshot 13 h 29 min after it was published. https://rattlesnakesbymail.com/observed/googlebot
- F-261118-03 GoogleOther first fetched the 2026-11-17 snapshot 14 h 46 min after it was published. https://rattlesnakesbymail.com/observed/googleother
- F-261118-04 An MCP client named agent-index first called identify_user_agent on 2026-11-18. https://rattlesnakesbymail.com/mcp
- F-261118-05 Every request on 2026-11-18 from Googlebot, Bingbot, GPTBot, OAI-SearchBot and PerplexityBot was verified against the operator's published method. https://rattlesnakesbymail.com/method


## Records selected at random for 2026-11-19

Ten records from the field guide are chosen at random each day. The selection and how it was drawn are listed in snapshot.json.

- https://rattlesnakesbymail.com/compare/bingbot-vs-oai-adsbot
- https://rattlesnakesbymail.com/compare/applebot-extended-vs-chatgpt-user
- https://rattlesnakesbymail.com/fields/observed_fetches_json
- https://rattlesnakesbymail.com/claims/24
- https://rattlesnakesbymail.com/claims/146
- https://rattlesnakesbymail.com/compare/claude-searchbot-vs-perplexity-user
- https://rattlesnakesbymail.com/compare/claudebot-vs-perplexity-user
- https://rattlesnakesbymail.com/claims/121
- https://rattlesnakesbymail.com/compare/chatgpt-user-vs-gptbot
- https://rattlesnakesbymail.com/claims/63


## Crawlers

| Crawler | Requests | Verified share | Distinct paths | 28-day median | Sweep day | First path | robots.txt | sitemap |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| [Googlebot](https://rattlesnakesbymail.com/observed/googlebot) | 58 | 100% | 34 | 44 | no | /crawlers/oai-searchbot | 16 | 0 |
| [Bingbot](https://rattlesnakesbymail.com/observed/bingbot) | 31 | 100% | 19 | 38 | no | /claims/66 | 1 | 0 |
| [GPTBot](https://rattlesnakesbymail.com/observed/gptbot) | 6 | 100% | 4 | 4 | no | /sitemap-index.xml | 0 | 5 |
| [OAI-SearchBot](https://rattlesnakesbymail.com/observed/oai-searchbot) | 4 | 100% | 1 | 6 | no | /robots.txt | 4 | 0 |
| [PerplexityBot](https://rattlesnakesbymail.com/observed/perplexitybot) | 3 | 100% | 3 | 0 | no | /robots.txt | 1 | 0 |
| [GoogleOther](https://rattlesnakesbymail.com/observed/googleother) | 2 | 100% | 1 | 0 | no | /observed/perplexity-user | 0 | 0 |
| [Reflectionbot](https://rattlesnakesbymail.com/observed/reflectionbot) | 2 | 100% | 2 | 0 | no | /robots.txt | 1 | 0 |

## MCP calls

| Client | Tool | Calls |
| --- | --- | --- |
| BrickBlueBot | get_snapshot | 1 |
| BrickBlueBot | identify_user_agent | 1 |
| BrickBlueBot | list_changes | 1 |
| BrickBlueBot | lookup_crawler | 1 |
| FaceabotProbe | list_changes | 1 |
| agent-index | get_snapshot | 1 |
| agent-index | identify_user_agent | 1 |
| agent-index | list_changes | 1 |
| kitscout-live-check | get_snapshot | 1 |
| kitscout-live-check | list_changes | 1 |
| kitscout-live-check | lookup_crawler | 1 |
| mcp-index | __mcp_index_nonexistent__ | 1 |
| rokmcp-probe | list_changes | 1 |
| verifymcp-probe | __verifymcp_auth_probe_7b769759e46896bc__ | 1 |

## Snapshot pickup

For each of the previous seven snapshots, the first fetch by each verified crawler, in minutes after publication, and the c tag on that fetch.

| Snapshot | Crawler | Minutes after publication | Tag |
| --- | --- | --- | --- |
| 2026-11-17 | Bingbot | 293 | none |
| 2026-11-17 | Googlebot | 809 | none |
| 2026-11-17 | GoogleOther | 886 | h |
| 2026-11-16 | OAI-SearchBot | 78 | none |
| 2026-11-16 | GPTBot | 78 | h |
| 2026-11-16 | Bingbot | 293 | none |
| 2026-11-16 | Googlebot | 1,894 | none |
| 2026-11-16 | GoogleOther | 2,244 | p |
| 2026-11-15 | OAI-SearchBot | 77 | none |
| 2026-11-15 | GPTBot | 77 | h |
| 2026-11-15 | Bingbot | 292 | none |
| 2026-11-15 | Googlebot | 866 | none |
| 2026-11-15 | GoogleOther | 1,479 | p |
| 2026-11-14 | OAI-SearchBot | 77 | none |
| 2026-11-14 | GPTBot | 77 | h |
| 2026-11-14 | Bingbot | 293 | none |
| 2026-11-14 | Googlebot | 1,135 | none |
| 2026-11-14 | GoogleOther | 2,995 | none |
| 2026-11-13 | OAI-SearchBot | 78 | none |
| 2026-11-13 | GPTBot | 78 | h |
| 2026-11-13 | Bingbot | 293 | none |
| 2026-11-13 | Googlebot | 628 | none |
| 2026-11-13 | GoogleOther | 881 | none |
| 2026-11-12 | Bingbot | 292 | none |
| 2026-11-12 | Googlebot | 840 | none |
| 2026-11-12 | OAI-SearchBot | 1,517 | none |
| 2026-11-12 | GPTBot | 1,517 | p |
| 2026-11-11 | Bingbot | 294 | none |
| 2026-11-11 | Googlebot | 1,532 | none |
| 2026-11-11 | GoogleOther | 1,670 | h |
| 2026-11-11 | OAI-SearchBot | 2,958 | none |
| 2026-11-11 | GPTBot | 2,958 | p |

## Open questions

- [Can robots.txt cleanly separate "allow this page for search or citation" from "block it for training," and does that distinction hold across every major AI platform?](https://rattlesnakesbymail.com/questions/eaaa86558b443873916376dc4d0546f60ee5d890631698c41768fb619582451c) (id 26, asked 5 times)
- [How can AI citation performance be measured accurately, given there is no equivalent of a Search Console report for citations?](https://rattlesnakesbymail.com/questions/e61346be3fef727db8cefe0257815575a03a33698acfb1e0059140169dbd35ee) (id 34, asked 5 times)
- [Was a specific document used in a model's training, fine-tuning, or evaluation?](https://rattlesnakesbymail.com/questions/97b24d1729c7f32e2e09cb297328a2f7977235c746960d243536ee8b9e4e494d) (id 38, asked 4 times)
- [Why wasn't a specific page cited by an AI answer instead of a competitor's page?](https://rattlesnakesbymail.com/questions/e08c55e6af0f8858dae0af2af4eb4caafe28dfb739f1ed4163e9cb0d954bb2ff) (id 7, asked 3 times)
- [How much do backlinks, brand mentions, structured data, reviews, forum discussions, publisher reputation, and content freshness each matter to being cited?](https://rattlesnakesbymail.com/questions/064f0033dab1091575ab29137f558668ff6193e891221987fd9523c31b4ff78f) (id 11, asked 3 times)

If you can answer one of these with a source, send a note: POST https://rattlesnakesbymail.com/notes with target question:<id>, where <id> is the number shown with the question, and body=<your answer and its source>.

## Ask for something this snapshot lacks

What data about web crawlers did you look for and not find? Search https://rattlesnakesbymail.com/search?q=your+question. Searches with no match are recorded as gaps. A gap is published at /questions only when an editor publishes it; nothing is published automatically. To suggest something this daily snapshot should include, send a note with target snapshot:261118.

To send it, POST https://rattlesnakesbymail.com/notes as a web form or as JSON with target=snapshot:261118 and body=<your note>. The same note can be sent with target_type=snapshot and target_id=261118. The optional author_claim field says who is writing: a person, a model or an agent.

## Links

- [Daily counts for every crawler](https://rattlesnakesbymail.com/observed)
- [The crawler records](https://rattlesnakesbymail.com/crawlers)
- [Method](https://rattlesnakesbymail.com/method)
- [Data](https://rattlesnakesbymail.com/data)
- [MCP endpoint (POST JSON-RPC)](https://rattlesnakesbymail.com/mcp)
- [Dataset repository](https://github.com/crank-box/rattlesnakesbymail-data)
- [All snapshots](https://rattlesnakesbymail.com/snapshots)
- [This snapshot as JSON](https://261118.rattlesnakesbymail.com/snapshot.json)
- [This snapshot as Markdown](https://261118.rattlesnakesbymail.com/snapshot.md)
- [This snapshot as CSV](https://261118.rattlesnakesbymail.com/snapshot.csv)
- [Facts as JSON](https://261118.rattlesnakesbymail.com/facts.json)
- [About this site, for automated readers](https://261118.rattlesnakesbymail.com/about-this-site.md)
- [Previous snapshot (2026-11-17)](https://261117.rattlesnakesbymail.com/?c=p)

Next snapshot: [2026-11-19](https://261119.rattlesnakesbymail.com/?c=p)
