Robots.txt Generator & Tester – SEO Tools by Lochan Yadav

⚡ Quick Presets

🔧 Rule Builder

⚙️ Global Settings

Sitemap URL (recommended)

Crawl Delay (seconds · optional)

🧪 URL Path Tester

💡 How to use: Enter a user-agent (e.g. Googlebot) and a path (e.g. /admin/) then click Test to see if your rules would block or allow that crawler.

📄 Generated robots.txt

# robots.txt will appear here as you build rules…

📚 Rules Quick Reference

Disallow: /admin/Block all bots from the /admin/ folder and everything inside it

Allow: /public/Explicitly allow access (useful after a broad Disallow)

Disallow: *.pdf$Block all PDF files across the site

Disallow: /*?*Block all URLs with query parameters

Disallow: /Block the entire site (use only on staging/dev environments)

💡 Remember: robots.txt only guides cooperative crawlers — it does NOT protect sensitive data. Use authentication for that. Also, Disallow does not remove URLs from Google's index; use a noindex meta tag for that.

🤖 Common Crawlers Reference

Googlebot
Google Search

Bingbot
Bing / Microsoft

AhrefsBot
Ahrefs SEO tool

SemrushBot
Semrush SEO tool

GPTBot
OpenAI crawler

ClaudeBot
Anthropic crawler

anthropic-ai
Anthropic AI

*
All crawlers