Robots.txt Generator
Create a robots.txt file with Allow, Disallow and sitemap rules for search engine bots.
What is robots.txt?
robots.txt is a text file in your site's root that tells search engine crawlers which pages to crawl and which to skip. Blocking unnecessary pages such as admin panels, internal search results and carts focuses your crawl budget on the pages that matter.
Generator features
- Separate rule groups for each bot (User-agent)
- Allow and Disallow paths
- Sitemap address
- Download the finished file
Be careful
A mistaken 'Disallow: /' can stop your whole site from being crawled. Test the file with the robots.txt report in Google Search Console before going live. robots.txt isn't a security measure; protect private pages with authentication.
How to use Robots.txt Generator
- Choose a User-agent for the rule group (* for all bots).
- Add allowed and disallowed paths.
- Type your sitemap URL.
- Download robots.txt and upload it to your site root.
Why use this tool?
Prepare an SEO-friendly robots.txt in minutes without syntax mistakes.
FAQ
Does Disallow hide a page from Google?
Not reliably. It stops crawling, but the page can still be indexed through links; use a noindex meta tag to keep it out.
Where should robots.txt be uploaded?
To the root of your domain, reachable at https://example.com/robots.txt.
Can I block AI crawlers?
Yes. Create separate User-agent groups for bots like GPTBot or Google-Extended and add Disallow: /.
Related tools
XML Sitemap Generator
Create a valid sitemap.xml from a list of URLs to submit to search engines.
Open toolMeta Tag Generator
Create title, description, robots, canonical and viewport meta tags with correct escaping.
Open toolCanonical URL Generator
Normalise a URL and create the rel="canonical" tag that prevents duplicate content.
Open toolHTTP Status Codes
Search every HTTP status code by number, name and meaning: 200, 301, 404, 500 and more.
Open tool