ROBOT
robots.txt Generator
Build a robots.txt file with crawl rules, blocked paths, and a sitemap declaration
Crawl policy
robots.txt
User-agent: * Disallow:
About this tool
robots.txt tells crawlers which parts of your site to skip and where your sitemap lives. A single typo can hide the entire site from search, so this generator emits only valid directives from the choices you make.
How to use
- Choose whether to allow or block all crawlers by default.
- Add the paths you want disallowed, one per line.
- Enter your sitemap URL.
- Copy the output and save it as /robots.txt at your domain root.
FAQ
Does Disallow remove a page from Google?
No. It only stops crawling. A blocked URL can still be indexed from external links without a snippet. Use a noindex meta tag to remove a page from the index.
Where does the file have to live?
Only at the domain root — https://example.com/robots.txt. Crawlers do not read it from a subdirectory.
Should I declare the sitemap here?
Yes. Google no longer accepts sitemap ping requests, so the robots.txt declaration plus Search Console is the standard discovery path.