Real-world examples of LLMS.txt files for different types of websites
Example LLMS.txt for a SaaS application with API and dashboard
# LLMS.txt - SaaS Platform User-agent: GPTBot Allow: / Allow: /blog Allow: /docs Disallow: /api Disallow: /dashboard Disallow: /admin User-agent: Claude-Web Allow: / Allow: /blog Allow: /docs Disallow: /api Disallow: /dashboard Crawl-delay: 10 Sitemap: https://example.com/sitemap.xml
LLMS.txt configuration for an online store
# LLMS.txt - E-commerce Store User-agent: GPTBot Allow: / Allow: /products Allow: /categories Disallow: /checkout Disallow: /cart Disallow: /account Disallow: /admin User-agent: PerplexityBot Allow: / Allow: /products Allow: /categories Disallow: /checkout Disallow: /account Sitemap: https://store.example.com/sitemap.xml
Open access for AI crawlers on a content-focused website
# LLMS.txt - Content Blog User-agent: * Allow: / Allow: /posts Allow: /categories Allow: /tags Allow: /authors Crawl-delay: 5 Sitemap: https://blog.example.com/sitemap.xml
LLMS.txt for a digital agency with portfolio
# LLMS.txt - Digital Agency User-agent: GPTBot Allow: / Allow: /portfolio Allow: /services Allow: /blog Disallow: /client-portal Disallow: /admin User-agent: Claude-Web Allow: / Allow: /portfolio Allow: /services Allow: /blog Sitemap: https://agency.example.com/sitemap.xml
Blocking all AI crawlers from accessing content
# LLMS.txt - Private Site User-agent: * Disallow: / # Block all AI crawlers from accessing this site
Full access for AI crawlers to help with documentation
# LLMS.txt - Documentation User-agent: * Allow: / # Full access for all AI crawlers # This helps AI assistants provide accurate documentation Sitemap: https://docs.example.com/sitemap.xml
Use our free generator to create a custom LLMS.txt file for your website
Generate LLMS.txt