About the Robots.txt Generator
A robots.txt file is four lines of text that can accidentally deindex an entire site, which is why generating it carefully beats copying one from a forum. This tool builds the file from a few choices and explains what each directive actually does — including the one thing robots.txt cannot do.
How to use the Robots.txt Generator
- Choose whether to allow all crawlers by default.
- Add the paths you want to block, such as admin or internal search.
- Add rules for specific user agents if you need to treat them differently.
- Add your sitemap URL, then copy the file to your site root.
Frequently asked questions
Does robots.txt keep a page out of Google?
No, and this is the most damaging misunderstanding about it. It stops crawling, not indexing — a blocked URL with inbound links can still appear in results without a snippet. Use a noindex meta tag to prevent indexing.
Where must the file live?
At the root of the domain, exactly at /robots.txt. Crawlers do not look anywhere else, and it applies per protocol, subdomain, and port.
What happens if I disallow everything?
Crawlers stop fetching your pages, existing entries gradually decay out of the index, and organic traffic disappears. A stray "Disallow: /" left over from staging is a genuinely common outage.
Should I block CSS and JavaScript?
No. Google renders pages to evaluate them, so blocking assets means it sees a broken layout and may judge the page poorly, especially on mobile.