SEO Tools
Robots.txt Generator
Build a readable robots.txt file from explicit allow, disallow and sitemap rules.
✓ Free
✓ No account
◉ Runs on this device
SEO Tools
Powering up the tool…
TOOL ROBOTS-TXT-G
What this tool does
About the robots.txt generator
Choose user agents and paths, preview the exact directives and download plain text. The generator starts conservatively and explains that robots rules are not access control.
How to use it
- Start blank or from the WordPress preset.
- Add user agents and their allow/disallow paths; the exact directives preview as you type.
- Add your sitemap URL — the one line every robots.txt should carry.
- Download the plain-text file and serve it at /robots.txt.
Useful for
- Write a correct robots.txt for a new site without memorising directive syntax.
- Add sitemap lines and per-crawler rules to an existing file cleanly.
- Block a staging path or duplicate-content section from crawling.
Straight answers
Robots.txt Generator FAQ
Does robots.txt keep pages private?
No — it is a polite request to crawlers, not access control, and the tool says this in bold. Anything genuinely private needs authentication; a disallowed URL can still be indexed if other sites link to it.
What is the safest starting point?
Allow everything and declare your sitemap. Disallow only what you can name a reason for — over-blocking is the more common and more expensive robots.txt mistake, which is why the generator starts conservative.