# AI Law Framework: robots.txt # # Policy: the public guidance on this site may be read, indexed and cited by search engines, # used by AI assistants to answer questions, and used to train models (Content-Signal below). # Precedent text and firm data are behind sign-in and are not served to crawlers; the private # paths below are closed as well. # # These templates have a markdown version at the same address plus .md (the home page is /index.md), # or send Accept: text/markdown: home, how it works, connect, privacy, terms, frameworks, practice, # obligations, jurisdictions, courts, the AI rules tracker, software profiles, guides, glossary, # knowledge management, the AI policy, precedents, and workflows. An index for agents is at /llms.txt. # Cases, news, best-of pages, the A–Z index, quick-check results, and audience pages do not. # Each crawler is named so the policy is explicit per purpose; a crawler that matches a named # group ignores the * group, so every group repeats the same rules. # OpenAI model training User-agent: GPTBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # ChatGPT search results User-agent: OAI-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # a person asking ChatGPT to open a page User-agent: ChatGPT-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # Anthropic model training User-agent: ClaudeBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # Claude search results User-agent: Claude-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # a person asking Claude to open a page User-agent: Claude-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # Perplexity search index User-agent: PerplexityBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # a person asking Perplexity to open a page User-agent: Perplexity-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # Gemini training and grounding (not Google Search) User-agent: Google-Extended Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # Apple model training User-agent: Applebot-Extended Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # Common Crawl, the open corpus many models train on User-agent: CCBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # Meta AI training and retrieval User-agent: meta-externalagent Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # Amazon search and AI answers User-agent: Amazonbot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # ByteDance AI products User-agent: Bytespider Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # DuckDuckGo AI-assisted answers User-agent: DuckAssistBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # a person asking Le Chat to open a page User-agent: MistralAI-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ # everyone else, including search engines User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Allow: /api/v1/openapi.json Allow: /api/v1/docs Allow: /api/search Disallow: /sign-in Disallow: /sign-up Disallow: /forgot-password Disallow: /reset-password Disallow: /auth/ Disallow: /api/ Disallow: /admin Disallow: /search Disallow: /d/ Disallow: /decide Disallow: /policy Disallow: /unsubscribe Disallow: /decision-files Disallow: /reviews Disallow: /work Disallow: /firm Disallow: /oauth Disallow: /md/ Sitemap: https://ailawframework.com/sitemap.xml Sitemap: https://ailawframework.com/sitemaps/software.xml Sitemap: https://ailawframework.com/sitemaps/practice.xml Sitemap: https://ailawframework.com/sitemaps/jurisdictions.xml Sitemap: https://ailawframework.com/sitemaps/courts.xml Sitemap: https://ailawframework.com/sitemaps/guides.xml Sitemap: https://ailawframework.com/sitemaps/news.xml Sitemap: https://ailawframework.com/sitemaps/cases.xml Sitemap: https://ailawframework.com/sitemaps/glossary.xml Sitemap: https://ailawframework.com/sitemaps/frameworks.xml Host: https://ailawframework.com