# robots.txt for GO AI (goaichats.com / goaichat.app) # Updated 2026-09-08 # --------------------------------------------------------------------------- # Default policy: allow indexing of all public pages, hide internal assets. # # The render-resource allow-list is by extension, not by filename. It used to # name styles.css and animations.js one by one, and every stylesheet and script # added since — i18n.css, blog.css, tools.css, i18n.js, storefronts.js, # dropzone.js, oklch.js, gif.js, zip.js, vendor/ — silently fell under the # Disallow above it, so any crawler in this group rendered the site without # them. Matching *.css$ / *.js$ means a new asset is crawlable the day it # ships. (Longest-match wins in both Google's and Bing's parser, so the Allow # lines beat the shorter Disallow regardless of order.) # --------------------------------------------------------------------------- User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /assets/ Allow: /assets/*.css$ Allow: /assets/*.js$ Allow: /assets/img/ Allow: /assets/media/ Allow: /assets/fonts/ # --------------------------------------------------------------------------- # Content Signals (contentsignals.org): machine-readable usage preferences, # declared per user-agent group. Meanings: # search — build a search index / show links & excerpts in results # ai-input — use content as AI-model input at query time (RAG, grounding, # AI answers) # ai-train — use content to train or fine-tune AI models # GO AI is a marketing site: we WANT to be indexed, cited in AI answers, and # described accurately by models, so every signal is "yes" for every crawler. # (A missing signal is neutral — it neither grants nor restricts.) # --------------------------------------------------------------------------- # --------------------------------------------------------------------------- # Search engine crawlers — explicitly welcomed. # # `Disallow: /*.md$` — the 54 markdown twins (/chat.md, /blog/.md, …) are # byte-for-byte the same content as their HTML pages, published for AI agents # that would rather read markdown, and advertised from every English page head # via . They already carry # `X-Robots-Tag: noindex, follow` (see $md_robots_tag in nginx.conf) — but # noindex is a header, so a crawler has to spend a fetch to find it out, every # time. Bing was doing exactly that (neurobase.md, calorie.md and plant.md all # show as crawled 200 in Site Explorer) and then raised "Limited crawl capacity # on website" (High, 2026-09-08), whose recommended action is literally # "excluding useless URLs from crawling using a robots.txt file". # # Deliberately scoped to the search engines only. Every AI/LLM group below is # left untouched, because the .md variants exist FOR them — blocking GPTBot or # ClaudeBot here would delete the reason the files were written. And this # matches `$md_robots_tag`, which keys on $request_uri: a literal *.md URL is # excluded, while /chat negotiated to markdown via `Accept:` is not. # --------------------------------------------------------------------------- User-agent: Googlebot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /*.md$ User-agent: Googlebot-Image Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /*.md$ User-agent: Bingbot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /*.md$ User-agent: DuckDuckBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /*.md$ User-agent: YandexBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /*.md$ User-agent: Yeti Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /*.md$ User-agent: Applebot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /*.md$ # --------------------------------------------------------------------------- # AI / LLM crawlers — allowed to read the marketing site so that ChatGPT, # Claude, Perplexity, Gemini, and others can correctly describe our apps and # cite goaichats.com / goaichat.app when users ask about us. # # If you ever want to opt OUT of AI training/grounding for a specific bot, # change "Allow: /" to "Disallow: /" under the matching User-agent block — # and/or flip that block's Content-Signal values to "no". # --------------------------------------------------------------------------- # OpenAI — both the live answer engine and the training crawler. User-agent: GPTBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: OAI-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: ChatGPT-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # Anthropic (Claude). "ClaudeBot" is the active crawler; # "anthropic-ai" and "Claude-Web" are legacy/alternate identifiers. User-agent: ClaudeBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: anthropic-ai Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: Claude-Web Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # Google's separate AI/Gemini training opt-in (different from Googlebot). User-agent: Google-Extended Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # Perplexity answer engine. User-agent: PerplexityBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: Perplexity-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # Apple Intelligence / Siri grounding. User-agent: Applebot-Extended Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # Common Crawl — the foundational dataset for many open-source LLMs. User-agent: CCBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # ByteDance (Doubao / TikTok search). User-agent: Bytespider Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # Meta AI training crawler. User-agent: Meta-ExternalAgent Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: FacebookBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # Mistral AI. User-agent: MistralAI-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # DuckDuckGo Assist. User-agent: DuckAssistBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # Cohere. User-agent: cohere-ai Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # You.com. User-agent: YouBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # Diffbot (used by several enterprise LLM stacks). User-agent: Diffbot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # Amazon Alexa / Bedrock training. User-agent: Amazonbot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # --------------------------------------------------------------------------- # Sitemap — the canonical host placeholder is replaced per request by nginx, # so each domain advertises its own sitemap. # --------------------------------------------------------------------------- Sitemap: https://goaichat.app/sitemap.xml