A robots.txt file is four lines of text that can accidentally deindex an entire site, which is why generating it carefully beats copying one from a forum. This tool builds the file from a few choices and explains what each directive actually does — including the one thing robots.txt cannot do.
No, and this is the most damaging misunderstanding about it. It stops crawling, not indexing — a blocked URL with inbound links can still appear in results without a snippet. Use a noindex meta tag to prevent indexing.
At the root of the domain, exactly at /robots.txt. Crawlers do not look anywhere else, and it applies per protocol, subdomain, and port.
Crawlers stop fetching your pages, existing entries gradually decay out of the index, and organic traffic disappears. A stray "Disallow: /" left over from staging is a genuinely common outage.
No. Google renders pages to evaluate them, so blocking assets means it sees a broken layout and may judge the page poorly, especially on mobile.
All free tools · Blog · About · Contact