robots.txt Generator
Build a correct robots.txt with allow/disallow rules, crawl-delay and sitemap directives using handy presets. Free and runs in your browser.
Build a clean robots.txt right in your browser. Nothing is uploaded — everything is generated locally.
Place this file at your site root so it is reachable at https://your-site.com/robots.txt
About robots.txt Generator
This robots.txt generator builds a syntactically correct robots.txt file from user-agent groups, each with its own Allow and Disallow paths and an optional Crawl-delay. Add as many groups as you need — a wildcard * group for all crawlers plus specific rules for individual bots — and list one or more Sitemap directives at the end.
Quick presets cover the common starting points: allow everything, block everything, a sensible WordPress configuration, and a block-list for common bots. Start from a preset, then adjust the paths to match your site structure.
The file is generated locally in your browser and never uploaded. Copy the output or download robots.txt, then place it at your site root so crawlers find it at https://your-site.com/robots.txt — that exact location is the only one search engines check.
How to use robots.txt Generator
- Optionally start from a preset: Allow all, Block all, WordPress, or Block common bots.
- Edit the user-agent for each group (for example * or Googlebot) and add Disallow and Allow paths.
- Set an optional Crawl-delay in seconds and add more groups for other crawlers if needed.
- Add your sitemap URLs so crawlers can discover them.
- Copy the result or download robots.txt, and upload it to your site root.