Robots.txt Generator
Build a complete robots.txt file for your website — user-agent groups, allow/disallow rules, crawl delays and sitemap directives, all with a live preview and coverage score, processed entirely in your browser.
Robots.txt Generator
Each group targets a specific crawler with its own Allow / Disallow rules and optional crawl delay.
Adds a Sitemap: directive at the end of the file so search engines can discover your sitemap.
Your generated robots.txt appears here
Add user-agent groups, allow/disallow rules and a sitemap URL above — the code updates live with a coverage score and recommendations.
How It Works
Three simple steps to build your robots.txt file.
Add user-agent groups
Pick a crawler (or all robots) and add Allow / Disallow rules with their paths.
Configure sitemap & delay
Optionally set a crawl delay and a Sitemap: URL so bots find your sitemap.
Copy, download & deploy
Review the live preview, follow the coverage score, then copy or download robots.txt.
FAQ
Common questions about robots.txt files.
What is a robots.txt file?
A robots.txt file tells search engine crawlers which pages or directories they can or cannot access on your website.
How do I use the generated robots.txt?
Copy the generated text and place it in the root directory of your website (e.g., https://example.com/robots.txt).
Should every group have rules?
Yes. Each user-agent group should include at least one Allow or Disallow rule, otherwise it has no effect.
Is my data stored anywhere?
No. All processing happens in your browser. Your data never leaves your device.