Robots.txt Generator
Generate a robots.txt file conforming to the Robots Exclusion Protocol with user-agent rules and sitemap.
Privacy first: Your files never leave your browser. All processing happens locally on your device.
Rules
Rule 1
Frequently Asked Questions
What is robots.txt?
Robots.txt is a text file placed at the root of a website that tells search engine crawlers which pages they can and cannot access.
Does robots.txt block indexing?
No, robots.txt only prevents crawling. To block indexing, use noindex meta robots tags or X-Robots-Tag HTTP headers.