# robots.txt — Aditya Mallah # Last updated: 2026-07-10 # Policy: fully open. All crawlers — search engines, AI search agents, # and AI training bots — may scrape, index, and use site content # (including for model training and fine-tuning). # === Default: open to everyone === User-agent: * Allow: / # Machine-readable surfaces (also linked from the site) # Sitemap: https://adityamallah.com/sitemap-index.xml # LLM index: https://adityamallah.com/llms.txt # LLM full content: https://adityamallah.com/llms-full.txt # RSS: https://adityamallah.com/rss.xml Sitemap: https://adityamallah.com/sitemap-index.xml # === AI training & foundation-model crawlers (explicitly allowed) === User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: OAI-AdsBot Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: Claude-Code Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-Web Allow: / User-agent: Google-Extended Allow: / User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-News Allow: / User-agent: Googlebot-Video Allow: / User-agent: Google-InspectionTool Allow: / User-agent: AdsBot-Google Allow: / User-agent: Mediapartners-Google Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: / User-agent: Bytespider Allow: / User-agent: TikTokSpider Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: FacebookBot Allow: / User-agent: cohere-ai Allow: / User-agent: cohere-training-data-crawler Allow: / User-agent: Diffbot Allow: / User-agent: DeepSeekBot Allow: / User-agent: PanguBot Allow: / User-agent: Amazonbot Allow: / User-agent: Amzn-SearchBot Allow: / User-agent: Amzn-User Allow: / User-agent: Omgilibot Allow: / User-agent: omgili Allow: / User-agent: Panscient Allow: / User-agent: iAskBot Allow: / User-agent: iaskspider Allow: / User-agent: Yandex Allow: / User-agent: YandexAdditional Allow: / User-agent: YandexAdditionalBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: DuckAssistBot Allow: / User-agent: YouBot Allow: / User-agent: PhindBot Allow: / User-agent: Andibot Allow: / User-agent: ExaBot Allow: / User-agent: LinkupBot Allow: / User-agent: TavilyBot Allow: / User-agent: Kagi-Fetcher Allow: / User-agent: Bravebot Allow: / User-agent: KunatoCrawler Allow: / User-agent: Bingbot Allow: / User-agent: BingPreview Allow: / User-agent: MSNBot-Media Allow: / User-agent: ia_archiver Allow: / User-agent: archive.org_bot Allow: / User-agent: PetalBot Allow: / User-agent: SemrushBot Allow: / User-agent: AhrefsBot Allow: / User-agent: DotBot Allow: / User-agent: MJ12bot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Baiduspider Allow: / User-agent: Sogou Allow: / User-agent: Slurp Allow: / User-agent: facebookexternalhit Allow: / User-agent: LinkedInBot Allow: / User-agent: Twitterbot Allow: / User-agent: xAI-Grok Allow: / User-agent: Grok Allow: /