This article provides an overview of all bots recognised by AI Scrape Protect, divided into the four categories used in the plugin. Each bot includes an explanation of what it does and whether you would want to block it. For more information about the plugin, visit the AI Scrape Protect plugin page.
Version: 5.1
Published: 18 September 2026
Search Engines
These bots are allowed by default. They are responsible for regular search engine indexing. You can disable them individually, but doing so will directly affect your visibility in the relevant search engine.
Bingbot
Microsoft’s primary web crawler for Bing Search. Blocking it will remove your site from Bing search results.
Google-PageSpeed
Used by Google PageSpeed Insights for performance analysis of your site.
Google-Site-Verification
Used by Google Search Console to verify ownership of your site.
Googlebot
Google’s primary web crawler, responsible for indexing your site in Google Search. Blocking it will remove your site from Google search results.
Googlebot-Image
Crawls images for Google Images. Blocking it will remove your images from Google Images.
Googlebot-News
Indexes news articles for Google News.
Lighthouse
Google’s automated tool for analysing performance, accessibility, and SEO.
AI Training
These bots collect web content to train AI models. They are blocked by default. Blocking via robots.txt is effective for most of these bots, though some may ignore robots.txt directives.
AI2Bot
Developed by the Allen Institute for AI for academic research and AI model development.
Ai2Bot-Dolma
A variant of AI2Bot used to collect data for the Dolma open dataset, which is used to train open-source language models.
AIMatrixCrawler
Crawls web content for AI matrix and machine learning training purposes.
Amazon-AI
Amazon’s AI-specific crawler used to collect web content in support of services such as Alexa and recommendation systems.
Amazonbot
Amazon’s crawler used for indexing and AI model training, including Amazon Nova and Alexa. Respects robots.txt.
AnthropicBot
A crawler operated by Anthropic to collect data for training Claude. Some reports indicate this bot may ignore robots.txt directives.
Applebot-Extended
Apple’s opt-out control token for AI training. Blocking this prevents your content from being used to train Apple Intelligence and other Apple generative AI models. Activity has surged significantly since the rollout of Apple Intelligence in 2026.
BorderxBot
AI crawler from Borderxlab used for AI search support, assistant retrieval, and e-commerce product crawling.
CCBot
Common Crawl’s web crawler. Builds a publicly available open dataset used as training data by most major language models. Respects robots.txt.
ChatGLM-Spider
Crawler used by Zhipu AI to collect training data for ChatGLM, their large language model.
ClaudeBot
Anthropic’s primary training crawler. Collects web content to train Claude. Respects robots.txt.
ClaudeResearchBot
Deployed by Anthropic’s research team to collect datasets specifically for studying safe and responsible AI systems.
cohere-ai
Cohere’s crawler, used to collect data for training natural language processing models.
cohere-training-data-crawler
A dedicated Cohere crawler focused specifically on collecting training datasets for their AI models.
DataForSeoBot
Crawls websites to collect large datasets used for SEO analysis, data resale, and AI model training.
DeepseekBot
Crawler used by DeepSeek to collect web data for training their AI language models.
Diffbot
An AI-powered web scraper that structures web data for knowledge graph construction and various AI applications.
ErnieBot
Baidu’s AI model crawler, used to collect training data for Ernie Bot, Baidu’s large language model.
FirecrawlAgent
A scraper-as-a-service platform used by many different clients and AI applications to extract structured web content. Respects robots.txt but is multi-tenant, meaning many different operators drive the same crawler.
FishBot
Crawler operated by Fish Audio for open source AI data collection. Does not respect robots.txt.
Google-CloudVertexBot
Google’s crawler associated with the Vertex AI platform. Used for collecting web content to support Google’s enterprise AI services and model training.
GPTBot
OpenAI’s primary training crawler. Collects web content to train GPT models such as ChatGPT. Respects robots.txt.
GPTBot-Preview
An experimental pre-release variant of GPTBot used by OpenAI to test new data collection methods for future models.
iAskAI-Crawler
Collects web content to generate answers for the iAsk.ai search and question-answering platform.
ImagesiftBot
Specialises in crawling and collecting images, potentially for AI training or visual content analysis.
img2dataset
A tool used to download and process large image datasets for AI and machine learning training.
KimiBot
Crawler operated by Moonshot AI, the Chinese company behind the Kimi large language model. Used for training data collection.
KomoBot
Used by Komo Search to gather data for AI-enhanced search and training purposes.
magpie-crawler
Focuses on content aggregation and building training datasets, primarily for social listening and AI purposes.
Meta-ExternalAgent
Meta’s AI training and indexing crawler for products across Facebook, Instagram, and WhatsApp. Not to be confused with facebookexternalhit, Meta’s link preview fetcher, which should not be blocked.
Meta-ExternalFetcher
Used by Meta to fetch external content for AI indexing purposes. Distinct from facebookexternalhit, which handles link previews and should not be blocked.
MistralAI-Training
Mistral AI’s dedicated training crawler, used to collect web content for training their open and commercial language models.
NavuBot
Crawls websites on behalf of customers to train AI models for them. Does not respect robots.txt.
Omgilibot / Omgili
Scrapes content from forums and discussion platforms, primarily for market research and AI training datasets.
PanguBot
Linked to Huawei’s Pangu AI models. Collects web data for large language model training.
PiplBot
Gathers information for Pipl’s people search and identity verification services.
sentibot
Likely used for sentiment analysis data collection and AI model training.
StableDiffusionBot
Crawls web content, particularly images, for training Stable Diffusion and other generative AI image models.
Timpibot
Indexes and fetches data for search and AI applications.
TurnitinBot
Used by Turnitin to collect content for plagiarism detection databases and AI training. Blocking it prevents your texts from being included in their detection systems.
WARDBot
An AI data scraper that collects web content for AI training and monitoring purposes.
webzio / Webzio-Extended
Crawlers from Webz.io used to gather web data for content analysis and AI training datasets.
YouBot
Crawler used by You.com to collect content for AI-powered search and model training.
AI Search & Answers
These bots retrieve content for AI-driven search results or direct answers to users. They are blocked by default, but allowing them may help your site appear in AI search engines such as ChatGPT Search or Perplexity.
ChatGPT-User
Fetches pages in real time when a ChatGPT user requests up-to-date information. Blocking this prevents your content from being retrieved for live ChatGPT answers.
Claude-SearchBot
Anthropic’s retrieval indexing crawler for Claude’s search capabilities. Separate from ClaudeBot, which is used for training.
Claude-User
Fetches pages in real time when a Claude user requests current information. Blocking this prevents your content from being retrieved for live Claude answers. Does not affect search engine indexing or ranking.
DuckAssistBot
DuckDuckGo’s bot for DuckAssist, their AI-driven answer and content summary feature.
Google-Agent
Google’s user-triggered fetcher, used when a user asks Gemini or Google AI Overviews for current information. Note: this bot does not respect robots.txt. Blocking via robots.txt is therefore ineffective.
Google-Extended
Google’s opt-out control token for AI training and Gemini products. Blocking this prevents your content from being used to train Google’s AI models and from appearing in AI-generated summaries, while leaving regular Google Search indexing unaffected.
GoogleOther
A Google crawler used for internal purposes such as AI research and training, not for regular search indexing. Blocking this has no effect on your Google Search rankings.
OAI-SearchBot
OpenAI’s retrieval crawler for ChatGPT Search. Separate from GPTBot. Allowing this bot while blocking GPTBot lets your site appear in ChatGPT Search without contributing to model training.
OpenAIContentCrawler
Gathers data specifically for OpenAI’s content-related tools and retrieval-augmented generation.
Perplexity-User
Fetches pages in real time when a user submits a query to Perplexity AI. Blocking this prevents your content from appearing in Perplexity answers.
PerplexityBot
Perplexity AI’s primary crawler used to build its retrieval index for AI-generated search answers. Known for high crawl frequency and a focus on news and high-authority content. Provides no referral traffic back to your site.
General Indexing
These bots are primarily used for search engine indexing or data collection, but are allowed by default. You can block them individually if you consider it appropriate.
360Spider
Web crawler used by Qihoo 360’s search engine in China. Also used for AI-related data collection.
AwarioRssBot
Awario’s RSS-focused crawler used to monitor RSS feeds for brand mentions and content updates.
AwarioSmartBot
Awario’s primary crawler for brand monitoring and social listening. Collects publicly available content to track brand mentions and online conversations.
Baiduspider
Baidu’s primary web crawler for indexing in Baidu Search, the dominant search engine in China. Also feeds Baidu’s AI products. Blocking it will remove your site from Baidu Search.
Bytespider
Associated with ByteDance, TikTok’s parent company. May collect data for AI and content generation tools. ByteDance has no official documentation page for this crawler.
FacebookBot
Used by Meta for indexing public content for social media search and discovery features. Not to be confused with facebookexternalhit, which handles link previews and should not be blocked.
Grok / GrokAI / XAI / XBot
Several user agent variants associated with xAI, the AI company behind Grok. Used for data collection and AI research. xAI has no official documentation page for these crawlers.
PetalBot
Huawei’s web crawler associated with Petal Search. Used for search engine indexing with potential AI data applications.
QualifiedBot
Crawler operated by Qualified, a B2B sales intelligence platform. Collects web content for AI-powered sales and marketing tools.
SemrushBot-SWA
Semrush’s site audit crawler. Used by Semrush customers and agencies to audit websites for SEO issues. Blocking this prevents third parties from auditing your site via Semrush tools.
Sogou
Sogou’s web crawler for Sogou Search, a major search engine in China now operated by Tencent. Also used for AI data collection.
YandexBot
Yandex’s primary web crawler for search indexing. Also used for AI applications within Yandex services. Blocking it will remove your site from Yandex Search.
For more information about how AI Scrape Protect works, visit the AI Scrape Protect plugin page.
“`
