How to Create an LLMS.txt File: Complete Tutorial

Creating an LLMS.txt file is straightforward. Follow this step-by-step guide to set up AI crawler control for your website.

Step 1: Plan Your Crawler Strategy

Before creating your LLMS.txt file, decide:

- Which AI crawlers should access your site?

- What content should be accessible?

- What should be restricted?

Step 2: Choose Your Method

Option A: Use Our Free Generator (Recommended)

1. Visit our [LLMS.txt Generator](/)

2. Select AI crawlers (GPTBot, Claude, Perplexity, etc.)

3. Configure allowed/disallowed paths

4. Set optional crawl delays

5. Download your file

Option B: Create Manually

Create a text file named `llms.txt` with this structure:

LLMS.txt - AI Crawler Rules

User-agent: GPTBot

Allow: /

Disallow: /admin

Disallow: /api

User-agent: Claude-Web

Allow: /

Disallow: /private

Crawl-delay: 10

Sitemap: https://yoursite.com/sitemap.xml

Step 3: Configure Rules

Allow Rules

Specify which paths AI crawlers CAN access:

Allow: /

Allow: /blog

Allow: /docs

Disallow Rules

Specify which paths AI crawlers CANNOT access:

Disallow: /admin

Disallow: /api

Disallow: /private

Crawl Delay

Set a delay (in seconds) between requests:

Crawl-delay: 10

Step 4: Upload to Your Website

1. Save the file as `llms.txt`

2. Upload to your website's root directory

3. Verify it's accessible at `https://yoursite.com/llms.txt`

Step 5: Test Your Configuration

- Visit `https://yoursite.com/llms.txt` in your browser

- Verify the content displays correctly

- Check your server logs for crawler activity

Common Configurations

Full Access (Public Site)

User-agent: *

Allow: /

Restricted Access (Private Site)

User-agent: *

Disallow: /

Selective Access (Most Common)

User-agent: *

Allow: /

Disallow: /admin

Disallow: /api

Disallow: /checkout

Best Practices

1. **Start permissive**: Allow most content, restrict only sensitive areas

2. **Be specific**: Use exact paths for better control

3. **Test thoroughly**: Verify rules work as expected

4. **Update regularly**: Review and update as your site changes

5. **Monitor logs**: Track crawler behavior

Need Help?

Use our [free LLMS.txt generator](/) to create a properly formatted file in seconds. No technical knowledge required!

[Generate LLMS.txt Now](/)