How to Block AI Bots and Crawlers From Your Website

Every page you publish is potential training data. AI companies run dedicated crawlers that read the open web to train and ground their models, and most of them honor an opt-out, if you know the right names. Here's how to block AI bots with robots.txt, what each crawler does, and how to enforce the rules when a bot won't take no for an answer.

The AI Crawlers Worth Knowing

  • GPTBot — OpenAI's training crawler.
  • ClaudeBot — Anthropic's crawler.
  • Google-Extended — controls whether Google uses your content for AI training. Separate from Googlebot, so blocking it doesn't touch your search rankings.
  • CCBot — Common Crawl, whose archives feed many training datasets.
  • PerplexityBot — crawler for the Perplexity answer engine.
  • Applebot-Extended — Apple's AI training opt-out, separate from regular Applebot.
  • Bytespider — ByteDance's crawler, widely reported as a heavy scraper.
  • meta-externalagent — Meta's AI training crawler.

The Copy-Paste robots.txt for AI Bots

Add these rules to the robots.txt at your domain root to opt out of all of the above:

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: meta-externalagent
Disallow: /

Want AI assistants to cite you but not train on you? Some crawlers separate the two roles. Keep the training bots blocked and leave search-oriented agents alone, and revisit the list a couple of times a year, because new crawlers keep appearing.

When robots.txt Isn't Enough

robots.txt is a posted sign, not a lock. Well-known AI companies generally respect it; anonymous scrapers don't. For enforcement:

  • CDN and firewall rules. Major CDNs (Cloudflare, for example) offer one-click blocking of known AI crawlers and bot-score rules that catch impersonators.
  • Server-level blocking. Match declared user agents in your web server config to return 403s, and rate-limit aggressive IP ranges.
  • Verify the big bots. Real GPTBot and Googlebot requests come from published IP ranges, so you can reject fakes claiming those names.

Blocking Bots Out, Detecting AI In

Blocking crawlers protects your content from training. The mirror problem is AI content flowing into your platform: user uploads, marketplace listings, profile photos. AI or Not provides API access to the same detection that powers AI Blocker, covering images, text, audio, and video at platform scale.

Blocking AI Bots FAQ

How do I block AI bots from my website?

Add disallow rules for AI crawlers like GPTBot, ClaudeBot, and CCBot to your robots.txt file, and enforce them with bot-blocking rules at your CDN or firewall. robots.txt alone is a request; the CDN layer makes it a wall.

Does robots.txt actually stop AI crawlers?

Major AI companies state that their named crawlers respect robots.txt, and compliance from them is generally good. robots.txt is voluntary, though: bots that ignore it can only be stopped with server or CDN-level blocking.

Will blocking AI bots hurt my Google rankings?

Not if you block the right bots. Google-Extended controls AI training use and is separate from Googlebot, which handles search indexing. Blocking GPTBot, CCBot, or ClaudeBot has no effect on search crawling either.

Can I detect AI-generated content submitted to my site?

Yes. If your platform accepts user uploads, AI or Not offers detection for images, text, audio, and video via API, the same detection that powers AI Blocker in the browser.

Related Guides