Robots.txt Generator
Generate a simple robots.txt file with user-agent, allow, disallow, and sitemap rules.
Robots.txt Generator
Create a simple robots.txt file for crawl guidance and sitemap discovery.
Result
User-agent: * Allow: / Disallow: /admin Disallow: /private Sitemap: https://example.com/sitemap.xml
What the tool does
The Robots.txt Generator creates a basic robots.txt file for crawl guidance. It lets you set the user agent, allowed paths, disallowed paths, and sitemap URL, then copies the finished file text.
Robots.txt files help crawlers understand which public areas of a site they should or should not crawl. They also provide a convenient place to link to the XML sitemap.
How it works
Enter a user agent, allowed paths, disallowed paths, and a sitemap URL. The generator formats those values into standard robots.txt lines. Multiple allow or disallow paths can be entered one per line.
The output is plain text. You can copy it into a file named robots.txt and place that file at the root of your site.
When to use it
Use this tool when launching a new website, adding a sitemap, creating a staging site, or cleaning up crawl rules. It is especially helpful when you want a simple starting point and do not need complex crawler-specific rules.
Before publishing, read robots.txt for beginners. If you are preparing page metadata too, use the meta tag generator.
Benefits
A clear robots.txt file can help crawlers discover your sitemap and avoid low-value or private-like sections that should not be crawled. It also gives teams a visible place to review crawl instructions.
The generator reduces formatting mistakes. Robots.txt syntax is simple, but misplaced slashes or missing labels can create confusion.
Examples
For a public site, you might allow "/" and disallow "/admin" and "/private". For a staging site, you might disallow "/" to discourage crawling before launch. For a normal content site, include the sitemap URL so crawlers can find important pages.
After publishing a robots.txt file, open it in the browser and read it like a crawler would. Check whether every important public section is allowed and whether private or low-value paths are handled intentionally. If you recently added a blog, tools directory, or sitemap route, make sure the file still supports that new structure.
Keep robots.txt changes in your normal launch checklist. A rule that made sense during staging can become harmful after launch. Review it whenever you add a new section, change URL patterns, move a sitemap, or migrate hosting providers.
If you are unsure about a rule, start conservatively. Allow important public content, disallow only paths you understand, and test the final file before submitting important URLs for indexing.
Never copy another site's robots.txt without adapting it to your own routes.
Frequently Asked Questions
Does robots.txt hide private content? No. It is not security. Use authentication for private pages.
Where should robots.txt live? It should be available at the domain root, such as "https://example.com/robots.txt".
Should I include a sitemap? Yes, if your site has one. It helps crawlers discover URLs.
What does User-agent: * mean? It means the rules apply to all crawlers that follow robots.txt.
Can a wrong rule block my whole site? Yes. "Disallow: /" asks crawlers not to crawl the entire site, so use it carefully.