ToolReign
SEO

Robots.txt Generator

Build a valid robots.txt file with allow/disallow rules, sitemap reference, and crawl-delay. Supports Googlebot, Bingbot, and all user-agents.

robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/

Sitemap: https://toolreign.comm/sitemap.xml

Related Tools

More free utilities you might find useful

🗺️
XML Sitemap Generator
Create sitemaps
🔗
Canonical Tag Generator
Canonical URLs
✅
Meta Tags Checker
Audit meta tags
{}
Schema Generator
JSON-LD markup

Frequently Asked Questions

Quick answers to common questions

What is robots.txt?+

robots.txt is a file at your site root that tells search engine crawlers which pages they can or cannot access. It does not block pages from being indexed if linked externally.

Where do I put robots.txt?+

Upload it to your website root: https://yoursite.com/robots.txt. It must be accessible at that exact path.

Should I disallow admin pages?+

Yes. Disallow /admin/, /api/, staging areas, and internal search results. Do not use robots.txt to hide sensitive data — use authentication instead.

Is my data private?+

Yes. Generated locally in your browser.