Robots.txt Generator

What is robots.txt?

robots.txt is a small text file at the root of a website (for example example.com/robots.txt). It tells search engine crawlers which parts of the site they may visit. It is a request, not a lock: well-behaved bots follow it, but it does not hide private pages from people. Protect private areas with a password instead.

How to use this generator

1. Choose whether search engines may crawl the whole site.
2. List folders or pages you do not want crawled, such as admin areas, carts, internal search results or test pages. Start each with a slash.
3. Add your sitemap address so search engines find all your pages faster.
4. Copy or download the file and upload it to the main folder of your website (usually public_html), so it opens at yourdomain.com/robots.txt.

Be careful

• Block the whole site removes your pages from Google over time. Use it only for test or staging sites.
• Do not block CSS, JavaScript or image folders that your pages need; Google uses them to understand the page.
• To keep a page out of search results, use a noindex meta tag on the page instead of blocking it here — Google must be able to crawl the page to see that tag.
• After uploading, check the file in Google Search Console.