Robots.txt Generator Pro
Generate a custom robots.txt file for your website. Configure crawler access, manage disallowed paths, add your sitemap URL, and download your configuration as a ready-to-use text file.
Generator Settings
Configure the rules for your robots.txt file.
Select a starting configuration. You can edit the generated rules afterward.
Enter your website's canonical domain. Include HTTPS if your website supports it.
Leave blank to omit the Sitemap directive. The sitemap must exist at the URL you enter.
The selected user-agent determines which crawler the rules address.
Add URL paths that should be explicitly allowed. Enter one path per line.
Add paths that crawlers should avoid. Rules are written exactly as entered.
Optional directives or comments. Enter one line per rule. Check crawler support before using non-standard directives.
Generated File
You can edit this output directly before copying or downloading it.
How to Use Robots.txt Generator
Create your configuration and publish it using your website platform's supported method.
1. Enter Your Domain
Enter your website's canonical domain and optionally provide the URL of your existing sitemap.
2. Configure Rules
Choose a preset and configure allow and disallow paths. Avoid blocking pages or resources that search engines need.
3. Download and Publish
Review the output, copy it or download robots.txt, then publish it at the root of your website using a supported method.
Common Robots.txt Rules
Understand the main directives before applying them to your website.
| Directive | Purpose | Example |
|---|---|---|
| User-agent | Specifies the crawler to which a group of rules applies. | User-agent: * |
| Allow | Allows crawling of a matching path, subject to crawler-specific rule handling. | Allow: /public/ |
| Disallow | Requests that a crawler covered by the rule avoid a matching path. | Disallow: /private/ |
| Sitemap | Provides a sitemap URL to crawlers that support this directive. | Sitemap: https://example.com/sitemap.xml |
| Comment | Adds a human-readable note that crawlers ignore as a rule. | # Website crawler rules |
Frequently Asked Questions
Learn the basics of robots.txt configuration and crawler access.
What is a robots.txt file?
A robots.txt file is a plain-text file placed at the root of a website. It communicates crawler access rules to bots that follow the Robots Exclusion Protocol.
For example, a website may use it to request that crawlers avoid certain URL paths.
Is this Robots.txt Generator free?
Yes. This generator runs in your browser and does not require an API key, paid subscription, or external generation service.
Does this tool automatically publish my robots.txt file?
No. The tool generates, copies, and downloads the text file. You must publish the configuration through your hosting provider or website platform.
Blogger has its own robots.txt settings in the blog settings area. Use that supported setting rather than assuming a downloaded file can be uploaded to Blogger's root directory.
Where should I place the robots.txt file?
For a conventional website, the file is normally served from the root of the
applicable host, such as https://example.com/robots.txt.
A robots.txt file applies to its host and protocol scope. A file at a subdirectory does not control the entire website.
What does User-agent: * mean?
The asterisk is a wildcard user-agent value. It identifies a group of rules intended for crawlers that do not have a more specific matching user-agent group. Individual crawlers may interpret or support directives differently.
Does Disallow prevent a page from appearing in Google Search?
Not necessarily. Disallow requests that a compliant crawler avoid crawling a URL. A blocked URL may still appear in search results if it is discovered through other sources.
Robots.txt is not a reliable way to keep confidential information private or guarantee that a page will be removed from search results.
Can robots.txt protect private pages or files?
No. Robots.txt is a crawler instruction file, not an access-control or security system. Use authentication, authorization, or server-side access restrictions to protect private information.
Should I add a sitemap URL?
Adding a sitemap directive can help supported crawlers discover your sitemap. Enter the actual, publicly accessible sitemap URL for your website.
Do not add a sitemap URL that does not exist or belongs to another website.
Can I use this generator for Blogger?
Yes. You can use it to prepare a robots.txt configuration for a Blogger website. Select the Blogger preset, review every rule, and then configure the supported custom robots.txt setting in Blogger if it is available for your blog.
Do not blindly paste unfamiliar rules into a live blog. The best configuration depends on your website structure and indexing requirements.
What is the difference between Allow and Disallow?
Disallow requests that a crawler avoid a matching path, while Allow can explicitly permit a matching path. Some crawlers compare matching rules and apply their own precedence logic, so verify the behavior for your target crawler.
Why does my downloaded file have a .txt extension?
Robots.txt is a plain-text file. The download button saves the generated content
as robots.txt, which is the conventional filename for crawler rules.
Will these rules work for every search engine?
Not every crawler supports every directive. Common directives such as User-agent and Disallow are widely used, but support and rule interpretation can vary. Check the documentation for each crawler you want to manage.
A browser-based utility for creating and downloading robots.txt configurations. Always review generated rules and test your live configuration before relying on it.