AI Crawlers are automated agents used by large language model (LLM) providers and AI search systems to discover, fetch, and index web content for training, grounding, or retrieval purposes. Unlike traditional search engine crawlers—which primarily index HTML pages for keyword-based ranking—AI Crawlers focus on extracting structured data, factual claims, and entity relationships to feed machine learning models. They often prioritize machine-friendly endpoints such as llms.txt, JSON-LD schema, and markdown mirrors, using this information to improve the accuracy, transparency, and source attribution of AI-generated answers.