table-of-bot-metrics.md

Name	Operator	Respects robots.txt	Data use	Visit regularity	Description
AI2Bot	Ai2	Yes	Content is used to train open language models.	No information provided.	Explores 'certain domains' to find web content.
Ai2Bot-Dolma	Ai2	Yes	Content is used to train open language models.	No information provided.	Explores 'certain domains' to find web content.
Amazonbot	Amazon	Yes	Service improvement and enabling answers for Alexa users.	No information provided.	Includes references to crawled website when surfacing answers via Alexa; does not clearly outline other uses.
anthropic-ai	Anthropic	Unclear at this time.	Scrapes data to train Anthropic's AI products.	No information provided.	Scrapes data to train LLMs and AI products offered by Anthropic.
Applebot	Unclear at this time.	Unclear at this time.	AI Search Crawlers	Unclear at this time.	Applebot is a web crawler used by Apple to index search results that allow the Siri AI Assistant to answer user questions. Siri's answers normally contain references to the website. More info can be found at https://darkvisitors.com/agents/agents/applebot
Applebot-Extended	Apple	Yes	Powers features in Siri, Spotlight, Safari, Apple Intelligence, and others.	Unclear at this time.	Apple has a secondary user agent, Applebot-Extended ... [that is] used to train Apple's foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.
Bytespider	ByteDance	No	LLM training.	Unclear at this time.	Downloads data to train LLMS, including ChatGPT competitors.
CCBot	Common Crawl Foundation	Yes	Provides open crawl dataset, used for many purposes, including Machine Learning/AI.	Monthly at present.	Web archive going back to 2008. Cited in thousands of research papers per year.
ChatGPT-User	OpenAI	Yes	Takes action based on user prompts.	Only when prompted by a user.	Used by plugins in ChatGPT to answer queries based on user input.
Claude-Web	Anthropic	Unclear at this time.	Scrapes data to train Anthropic's AI products.	No information provided.	Scrapes data to train LLMs and AI products offered by Anthropic.
ClaudeBot	Anthropic	Yes	Scrapes data to train Anthropic's AI products.	No information provided.	Scrapes data to train LLMs and AI products offered by Anthropic.
cohere-ai	Cohere	Unclear at this time.	Retrieves data to provide responses to user-initiated prompts.	Takes action based on user prompts.	Retrieves data based on user prompts.
Diffbot	Diffbot	At the discretion of Diffbot users.	Aggregates structured web data for monitoring and AI model training.	Unclear at this time.	Diffbot is an application used to parse web pages into structured data; this data is used for monitoring or AI model training.
DuckAssistBot	Unclear at this time.	Unclear at this time.	AI Assistants	Unclear at this time.	DuckAssistBot is used by DuckDuckGo's DuckAssist feature to fetch content and generate realtime AI answers to user searches. More info can be found at https://darkvisitors.com/agents/agents/duckassistbot
FacebookBot	Meta/Facebook	Yes	Training language models	Up to 1 page per second	Officially used for training Meta "speech recognition technology," unknown if used to train Meta AI specifically.
FriendlyCrawler	Unknown	Yes	We are using the data from the crawler to build datasets for machine learning experiments.	Unclear at this time.	Unclear who the operator is; but data is used for training/machine learning.
Google-Extended	Google	Yes	LLM training.	No information.	Used to train Gemini and Vertex AI generative APIs. Does not impact a site's inclusion or ranking in Google Search.
GoogleOther	Google	Yes	Scrapes data.	No information.	"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development."
GoogleOther-Image	Google	Yes	Scrapes data.	No information.	"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development."
GoogleOther-Video	Google	Yes	Scrapes data.	No information.	"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development."
GPTBot	OpenAI	Yes	Scrapes data to train OpenAI's products.	No information.	Data is used to train current and future models, removed paywalled data, PII and data that violates the company's policies.
iaskspider/2.0	iAsk	No	Crawls sites to provide answers to user queries.	Unclear at this time.	Used to provide answers to user queries.
ICC-Crawler	NICT	Yes	Scrapes data to train and support AI technologies.	No information.	Use the collected data for artificial intelligence technologies; provide data to third parties, including commercial companies; those companies can use the data for their own business.
ImagesiftBot	ImageSift	Yes	ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support our suite of web intelligence products	No information.	Once images and text are downloaded from a webpage, ImageSift analyzes this data from the page and stores the information in an index. Our web intelligence products use this index to enable search and retrieval of similar images.
img2dataset	img2dataset	Unclear at this time.	Scrapes images for use in LLMs.	At the discretion of img2dataset users.	Downloads large sets of images into datasets for LLM training or other purposes.
ISSCyberRiskCrawler	ISS-Corporate	No	Scrapes data to train machine learning models.	No information.	Used to train machine learning based models to quantify cyber risk.
Kangaroo Bot	Unclear at this time.	Unclear at this time.	AI Data Scrapers	Unclear at this time.	Kangaroo Bot is used by the company Kangaroo LLM to download data to train AI models tailored to Australian language and culture. More info can be found at https://darkvisitors.com/agents/agents/kangaroo-bot
Meta-ExternalAgent	Meta	Yes.	Used to train models and improve products.	No information.	"The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly."
Meta-ExternalFetcher	Unclear at this time.	Unclear at this time.	AI Assistants	Unclear at this time.	Meta-ExternalFetcher is dispatched by Meta AI products in response to user prompts, when they need to fetch an individual links. More info can be found at https://darkvisitors.com/agents/agents/meta-externalfetcher
OAI-SearchBot	OpenAI	Yes	Search result generation.	No information.	Crawls sites to surface as results in SearchGPT.
omgili	Webz.io	Yes	Data is sold.	No information.	Crawls sites for APIs used by Hootsuite, Sprinklr, NetBase, and other companies. Data also sold for research purposes or LLM training.
omgilibot	Webz.io	Yes	Data is sold.	No information.	Legacy user agent initially used for Omgili search engine. Unknown if still used, omgili agent still used by Webz.io.
PanguBot	the Chinese company Huawei	Unclear at this time.	AI Data Scrapers	Unclear at this time.	PanguBot is a web crawler operated by the Chinese company Huawei. It's used to download training data for its multimodal LLM (Large Language Model) called PanGu. More info can be found at https://darkvisitors.com/agents/agents/pangubot
PerplexityBot	Perplexity	No	Used to answer queries at the request of users.	Takes action based on user prompts.	Operated by Perplexity to obtain results in response to user queries.
PetalBot	Huawei	Yes	Used to provide recommendations in Hauwei assistant and AI search services.	No explicit frequency provided.	Operated by Huawei to provide search and AI assistant services.
Scrapy	Zyte	Unclear at this time.	Scrapes data for a variety of uses including training AI.	No information.	"AI and machine learning applications often need large amounts of quality data, and web data extraction is a fast, efficient way to build structured data sets."
Sidetrade indexer bot	Sidetrade	Unclear at this time.	Extracts data for a variety of uses including training AI.	No information.	AI product training.
Timpibot	Timpi	Unclear at this time.	Scrapes data for use in training LLMs.	No information.	Makes data available for training AI models.
VelenPublicWebCrawler	Velen Crawler	Yes	Scrapes data for business data sets and machine learning models.	No information.	"Our goal with this crawler is to build business datasets and machine learning models to better understand the web."
Webzio-Extended	Unclear at this time.	Unclear at this time.	AI Data Scrapers	Unclear at this time.	Webzio-Extended is a web crawler used by Webz.io to maintain a repository of web crawl data that it sells to other companies, including those using it to train AI models. More info can be found at https://darkvisitors.com/agents/agents/webzio-extended
YouBot	You	Yes	Scrapes data for search engine and LLMs.	No information.	Retrieves data used for You.com web search engine and LLMs.