What is BotSpotter?

Every day, thousands of AI bots crawl the internet. Some are collecting pages to train the next big model. Some are indexing the web for AI search answers. Some are fetching a page right now because a person just asked ChatGPT or Claude a question. Most websites never see any of it.

BotSpotter makes it visible. A network of volunteer websites and honeypots reports every bot visit here. We count them live, put them on the map, and give each bot a name that stays the same wherever it shows up — so Copper Heron #0042 is the same bot on your blog, on a news site, and on a server in Singapore. When a bot nobody has seen before appears on three independent sites, it gets discovered and named here first.

Three kinds of AI bot

• Training crawlers — bulk-read the web to train models (GPTBot, ClaudeBot, Bytespider, Common Crawl).
• AI search — index pages so an AI can answer with citations (OAI-SearchBot, PerplexityBot).
• AI agents — fetch a page on behalf of a live user, this second (ChatGPT-User, Claude-User).

Plus the unidentified ones: automation that hasn't announced itself. Those are the interesting ones. They get names too.

Why it matters

• If you run a website: find out which AI companies are reading your pages and how often — free, two-minute install, no code for Nginx/Apache. Add your site as a sensor and climb the leaderboard.
• If you're curious: watch the AI economy happen in real time. Which countries are crawled most? Whose bots are busiest? What showed up this week that didn't exist last week?
• If you write or research: a public, live, citable source on AI crawler activity that isn't owned by a single cloud provider.

Run a bot? Claim it

Operators can claim their bot: prove control (make the bot fetch a one-time URL, or publish a token on its homepage), then give it a proper name, description and category. Claimed bots carry a ✓ and the generated name stays as an alias.

What we don't do

We don't store IP addresses, page contents, or anything about human visitors beyond a daily count. Bots are named by a hash, never exposed. Humans are counted without cookies. The numbers here are visits seen by participating sites, not “the whole internet” — the more sensors join, the closer it gets.

Add your site as a sensor

A sensor tells BotSpotter which bots visit your site. It never sends page content, cookies, header values, or your human visitors — browsers are filtered out on your server before anything leaves it. You get a public spot on the leaderboard and a private view of which AI companies read your pages.

1 · Get a sensor key

Keys are issued by hand while we're in beta. Fill this in and you'll get a key by email, usually the same day.

2 · Install (pick one)

Python app (FastAPI, Starlette, Flask, Django) — zero dependencies:

pip install botspotter-sensor

# FastAPI / Starlette
from botspotter_sensor import BotSpotterMiddleware
app.add_middleware(BotSpotterMiddleware, key="YOUR_KEY")

# Flask / any WSGI
from botspotter_sensor import wsgi_sensor
app.wsgi_app = wsgi_sensor(app.wsgi_app, key="YOUR_KEY")

# Django settings.py
MIDDLEWARE += ["botspotter_sensor.django.BotSpotterMiddleware"]
BOTSPOTTER_KEY = "YOUR_KEY"

Nginx / Apache / Caddy — no code, via Vector: add our log format, point Vector at the log file, set BS_KEY. Config files and a step-by-step are in sensors/ of the repo.

Anything else — POST JSON batches to /v1/ingest with header X-BS-Key:

{"events":[{"ua":"Mozilla/5.0 (compatible; GPTBot/1.2)",
            "headers":["host","user-agent","accept","accept-encoding"],
            "ip":"20.171.207.5","cf_country":"US","method":"GET","status":200,"path":"/robots.txt"}]}

3 · Show it off (and help others find BotSpotter)

Every sensor gets a live badge. Paste this where your site id is yours:

<a href="https://botspotter.app/?s=YOUR_SITE_ID"><img src="https://botspotter.app/badge/YOUR_SITE_ID.svg" alt="AI bots spotted here, by BotSpotter"></a>

← the portal's own badge, live.

4 · Watch it light up

Within a few minutes your site appears under Sensors with most catches, and every bot it sees gets its name. Behind Cloudflare? Nothing extra to do.

Sensor Terms & Privacy

Plain-English summary first; the terms follow. Effective 25 Oct 2026. These apply to anyone who installs a BotSpotter sensor or uses the BotSpotter API. Just looking at the site? Only the Privacy section concerns you.

Summary

• You install the sensor on sites you own or are authorised to administer. You're responsible for what you install.
• The sensor sends us bot-request metadata only; we discard IP addresses on arrival and never store page content.
• The service is free, provided as-is, with no uptime or accuracy guarantees.
• Aggregated data we publish is open under CC BY 4.0. Your site id appears publicly on the leaderboard.
• Either side can stop at any time: remove the sensor, or we revoke the key.

1. Who we are

BotSpotter (“we”, “us”) is an independent, non-commercial project operated by Bhushan Ladde, California, USA. Contact: [email protected].

2. Your responsibilities as a sensor operator

• You confirm you own, or are authorised by the owner to administer, every site on which you install a sensor.
• You are responsible for complying with laws that apply to you and your site, including any obligation to disclose third-party processing in your own privacy policy. We supply the description in §5 for that purpose.
• You will not modify the sensor to send data beyond what is described in §5, send data about sites you don't control, or send fabricated traffic.
• You keep your sensor key confidential. Anything sent with your key is attributed to your site.

3. What we provide

A free service that receives bot-request metadata from sensors, classifies and names bots, and publishes aggregated statistics. We may change, suspend, or discontinue any part of it at any time. We may revoke a key at our discretion, including for suspected abuse or fabricated data.

4. The software

The sensor code and the BotSpotter platform are open source under the MIT License. The license governs the software; these terms govern the service.

5. Data: what the sensor sends and what we keep

Sent by the sensor, per request your server judges to be automated: the names (not values) of the request headers in order; the User-Agent string; the client IP address; the CF-IPCountry header if present; HTTP method and status code; the timestamp; and a flag for whether the path was robots.txt. Requests that look like ordinary browsers are filtered on your server and not sent. No page content, cookies, header values, query strings, form data, or other paths are sent.

Processed on arrival and then discarded: the IP address. We use it in memory to derive (a) a country and approximate region, and (b) for unidentified bots only, a coarse network prefix (/16, up to 65,536 addresses) that becomes part of the bot's fingerprint hash. The address itself is never written to disk, logs, or backups.

Stored: timestamp, your site id, the bot's fingerprint hash and derived name, a version-stripped User-Agent family, bot operator and category, country, region, a coarse geohash, and the robots.txt flag. Raw events are kept 90 days; hourly aggregates indefinitely.

Human visitors to this portal: we count unique visitors using salted hashes of IP and User-Agent that are fed only into HyperLogLog counters (a probabilistic structure that cannot be reversed or enumerated). The daily hash includes the date and powers “today” and “this week”; a second hash without the date powers “all time” so a returning person is counted once. We also count visits per country and per referring domain (e.g. linkedin.com, “direct”) — never the referring page or path. No per-visitor record is written anywhere. No cookies, no local storage identifiers, no third-party analytics scripts.

Legal basis (where GDPR/UK GDPR applies): legitimate interest in measuring automated traffic. We have designed the processing so that no personal data is retained; the IP address is processed transiently for seconds. If you believe a stored record identifies you, email us and we will remove it.

6. Published data and your site id

Your site id (the name you choose when requesting a key) appears publicly on the sensor leaderboard with its hit counts. Choose a name you're comfortable publishing. Aggregated statistics and bot profiles we publish are licensed under CC BY 4.0; attribute “BotSpotter”.

7. No warranty

The service and data are provided “as is” and “as available”, without warranties of any kind, express or implied, including accuracy, completeness, merchantability, fitness for a particular purpose, or uninterrupted operation. Bot classification is heuristic and may be wrong. Do not use BotSpotter as the sole basis for blocking, billing, or legal decisions.

8. Limitation of liability

To the fullest extent permitted by law, we are not liable for any indirect, incidental, consequential, special, or punitive damages, or for lost profits, data, or goodwill, arising from the service or sensor, even if advised of the possibility. Our total liability for any claim is limited to USD 100 or the amount you paid us (zero), whichever is greater.

9. Indemnity

You will defend and hold us harmless from claims arising from your installation or use of a sensor on a site you were not authorised to administer, your breach of these terms, or your violation of applicable law.

10. Termination

You may stop at any time by removing the sensor. We may revoke keys or remove site ids at any time. Sections 5–11 survive termination. Historical aggregates already published remain published.

11. Governing law and changes

These terms are governed by the laws of the State of California, USA, without regard to conflict-of-law rules; courts in Sacramento County, California have exclusive jurisdiction. We may update these terms; material changes will be dated here and announced on the site. Continued use after a change is acceptance.

12. Claiming a bot

A bot may be claimed by its operator after verification (a fetch challenge the bot itself answers, or a token published on the bot's homepage domain). We store the contact email you give us as a salted hash once verified, plus the public fields you set (name, description, homepage, operator, category). You warrant that you operate the bot and that the information is accurate. Names must not impersonate others, mislead about the operator, or be offensive; we may edit or revoke a claim, and will do so on a substantiated dispute sent to [email protected]. The generated name remains a permanent public alias. The owner key is shown once; we cannot recover it, but you can re-verify to obtain a new one.

← add your site  ·  live counter

Claim your bot

Run a crawler or an agent that shows up here? Prove it's yours and you can give it a proper name, a description, a homepage and the right category. The generated name stays as a permanent alias so old links keep working. Claiming is free.

1 · Which bot?

Already verified?

Open /claim?slug=<your-bot> and paste your owner key: Manage

Abuse, impersonation or misleading names get the claim revoked (Terms §12). To dispute a claim, email [email protected].