What is the difference between Allow and Disallow in robots.txt?

Disallow tells a crawler it should not crawl a specific path. Allow tells a crawler it may crawl a path even if a broader Disallow rule would otherwise block it. Allow is used to create exceptions to Disallow rules. For example, Disallow: /admin/ blocks all of /admin/, while Allow: /admin/public/ would permit crawling of that specific sub-path despite the broader block. When Allow and Disallow rules conflict, the more specific rule (longer path) takes precedence for Googlebot.

What robots.txt file should I use for WordPress?

A recommended WordPress robots.txt file blocks admin pages, login pages, search results and plugin/theme files that should not be indexed. Typical WordPress robots.txt includes: Disallow: /wp-admin/ (allow /wp-admin/admin-ajax.php for AJAX), Disallow: /wp-login.php, Disallow: /?s= (search results), Disallow: /search, and Disallow: /trackback. Always allow /wp-content/uploads/ so Google can crawl your images, and allow /wp-content/themes/ and /wp-content/plugins/ so Google can render CSS and JavaScript correctly.

Robots.txt Generator

Q: What is a robots.txt file?

A robots.txt file is a plain text file placed at the root of a website (e.g. https://example.com/robots.txt) that instructs search engine crawlers which pages or sections of the site they should or should not crawl. It follows the Robots Exclusion Protocol (REP). Robots.txt does not prevent pages from being indexed if they are linked from other crawlable pages - it only controls crawler access. To prevent indexing, use a noindex meta tag instead.

Q: Does robots.txt affect SEO?

Yes, robots.txt affects SEO significantly. Blocking crawlers from important pages can prevent those pages from being indexed and ranking in search results. Conversely, accidentally blocking CSS and JavaScript files can prevent Google from rendering pages correctly, harming rankings. You should never block pages you want indexed. Robots.txt is best used to block duplicate content, admin pages, staging environments, search result pages and other content that should not appear in search results.

Q: What is a User-agent in robots.txt?

The User-agent directive specifies which crawler the following rules apply to. Use * (asterisk) to apply rules to all crawlers. Use a specific crawler name like Googlebot, Bingbot or GPTBot to apply rules only to that crawler. You can have multiple User-agent sections in a robots.txt file, each with different rules. Googlebot respects rules in the * section if no specific Googlebot section exists.

Q: Where should I put the robots.txt file?

The robots.txt file must be placed at the root of your domain, accessible at https://yourdomain.com/robots.txt. It cannot be placed in a subdirectory. The file must be publicly accessible without redirects and must return a 200 HTTP status code. For subdomains, each subdomain needs its own robots.txt file (e.g. blog.example.com/robots.txt is separate from example.com/robots.txt).

Q: What is the Sitemap directive in robots.txt?

The Sitemap directive in robots.txt tells search engines the location of your XML sitemap file. Add one or more Sitemap lines with the full URL to your sitemap. For example: Sitemap: https://example.com/sitemap.xml. This is not part of the original Robots Exclusion Protocol but is supported by Google, Bing and other major search engines. Adding your sitemap URL in robots.txt helps crawlers discover all your pages efficiently.

Q: What is Crawl-delay in robots.txt?

The Crawl-delay directive suggests how many seconds a crawler should wait between requests to your server. For example, Crawl-delay: 10 asks crawlers to wait 10 seconds between requests. Note that Googlebot does not respect Crawl-delay - it manages crawl rate automatically. You can control Googlebot's crawl rate in Google Search Console instead. Crawl-delay is respected by some other crawlers like Bingbot.

Key features

Everything the Robots.txt Generator Does

🤖

Visual Rule Builder

Add Allow, Disallow and Crawl-delay rules through a clean point-and-click interface - no manual syntax needed.

✅

Google Validation

Real-time validation warns about conflicting rules, incorrect Disallow: / usage and Googlebot-incompatible directives.

📋

6 Platform Presets

Load instant presets for WordPress, Shopify, generic eCommerce, block all, allow all and block AI crawlers.

🤖

Multiple User Agents

Add separate rule sections for Googlebot, Bingbot, GPTBot and any other crawler with different instructions per agent.

📍

Sitemap Declaration

Add one or more Sitemap: directives so search engines can discover your XML sitemap directly from robots.txt.

👁

Syntax-Highlighted Preview

Live preview renders your robots.txt with colour-coded directives - purple for User-agent, green for Allow, red for Disallow.

⬇

One-Click Download

Download the finished robots.txt file directly to your computer, ready to upload to your website root.

📋

Copy to Clipboard

Copy the complete robots.txt output with one click for pasting directly into your server or CMS.

💡

Crawl-delay Support

Add Crawl-delay directives for crawlers that respect them, with a note that Googlebot ignores this directive.

How to use

How to Use the Robots.txt Generator

Choose a preset

Select WordPress, Shopify, block AI crawlers or another platform preset from the dropdown to load a recommended starting configuration instantly.

Review user-agent sections

Each User-agent block controls which crawler the following rules apply to. Use * for all crawlers or enter a specific name like Googlebot.

Add or edit rules

Click + Add Rule inside any user-agent section. Choose Disallow (block), Allow (permit) or Crawl-delay and enter the path or value.

Add your sitemap URL

In the Sitemap URLs section, enter the full URL of your XML sitemap so crawlers can discover all your pages efficiently.

Check the validation panel

Review the validation panel on the right. Fix any red errors before downloading. Yellow warnings explain Googlebot behaviour differences.

Download your robots.txt

Click Download robots.txt to save the file to your computer, then upload it to the root directory of your website at yourdomain.com/robots.txt.

Competitor comparison

Robots.txt Generator: LazyTools vs Competitors

See how LazyTools compares to other popular tools. Our free robots.txt generator is the only option that combines all key features with no login required and complete browser-side privacy.

Feature	LazyTools	Google Search Console	Yoast SEO Plugin	seoptimer.com
Visual rule builder	Yes	No (text only)	Basic	No
Validation warnings	Yes (real-time)	Basic	No	No
Platform presets	Yes (6 presets)	No	WordPress only	No
Block AI crawlers preset	Yes	No	No	No
Multiple user agents	Yes	Yes	Limited	No
Sitemap directive	Yes	Yes	Yes	Yes
Download file	Yes	No	Yes (via plugin)	Yes
No login required	Yes	Requires account	Requires WordPress	Yes

Syntax guide

Robots.txt Syntax and Directives Explained

Directive	Syntax	Effect	Googlebot support
User-agent	User-agent: *	Specifies which crawler the following rules apply to. * means all crawlers.	Yes
Disallow	Disallow: /path/	Tells the crawler not to crawl this path or URL. Empty value means allow all.	Yes
Allow	Allow: /path/	Creates an exception to a Disallow rule. More specific path wins.	Yes
Sitemap	Sitemap: https://example.com/sitemap.xml	Tells crawlers the location of your XML sitemap. Can appear multiple times.	Yes
Crawl-delay	Crawl-delay: 10	Requests crawlers wait N seconds between requests. Googlebot ignores this.	No (use Search Console)
# Comment	# This is a comment	Lines starting with # are ignored by crawlers. Useful for documentation.	Ignored

How Allow and Disallow interact

When an Allow and a Disallow rule conflict, the more specific rule wins (the one with the longer path). For example, if you have Disallow: /admin/ and Allow: /admin/public/, Googlebot will not crawl anything in /admin/ except /admin/public/. If two rules are the same length, Googlebot uses the Allow rule.

# Correct: Allow overrides Disallow for a specific sub-path User-agent: * Disallow: /admin/ Allow: /admin/public/ # This works -- longer path wins # Common mistake: Disallow: / blocks EVERYTHING User-agent: * Disallow: / # Blocks all pages -- entire site hidden from crawlers

What robots.txt cannot do

Robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if it is linked from other crawlable pages, because Google can infer the page exists without crawling it. To prevent a page from being indexed, use a noindex meta tag or X-Robots-Tag HTTP header instead. Additionally, robots.txt only applies to well-behaved crawlers that follow the protocol. Malicious bots ignore it entirely.

Platform templates

Robots.txt Templates for WordPress and Shopify

WordPress robots.txt

This robots.txt generator for WordPress produces a recommended configuration. WordPress sites should block admin pages, search results and duplicate parameter URLs while allowing Google to access theme and plugin CSS and JavaScript files. Load the WordPress preset in the generator above to get a fully configured starting point.

# WordPress recommended robots.txt User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-login.php Disallow: /?s= Disallow: /search Disallow: /trackback Disallow: */feed Allow: /wp-content/ Sitemap: https://example.com/sitemap.xml

Shopify robots.txt

The robots.txt generator for Shopify preset creates a recommended configuration. Shopify automatically generates a robots.txt file for your store. However, you can customise it via the Shopify admin under Online Store > Preferences > robots.txt. Key rules for Shopify block checkout pages, account pages and internal search results. Load the Shopify preset above to see the recommended configuration.

Blocking AI web crawlers

Use this robots txt template to block Googlebot-Extended, GPTBot and other AI training crawlers. AI companies use web crawlers to collect training data. GPTBot (OpenAI), Claude-Web (Anthropic), CCBot (Common Crawl) and other AI crawlers can be blocked using their User-agent names. The generator's Block AI Crawlers preset adds rules for the major AI training crawlers. Note that not all AI crawlers respect robots.txt.

FAQ

Frequently Asked Questions

What is a robots.txt file?▼

A robots.txt file is placed at the root of a website (e.g. https://example.com/robots.txt) and instructs search engine crawlers which pages to crawl or skip. It follows the Robots Exclusion Protocol. Robots.txt controls crawling, not indexing - to prevent indexing use a noindex meta tag.

What is the difference between Allow and Disallow?▼

Disallow blocks a crawler from a path. Allow creates an exception to a Disallow rule. When rules conflict, the more specific (longer) path wins. Use Allow to permit a sub-path that is inside a broader Disallow block, for example allowing /wp-admin/admin-ajax.php while blocking /wp-admin/.

Does robots.txt affect SEO?▼

Yes significantly. Blocking important pages prevents them from being indexed and ranking. Accidentally blocking CSS and JavaScript files stops Google from rendering pages correctly, harming rankings. Never block pages you want indexed. Use robots.txt only for admin pages, duplicate content, staging environments and search result pages.

What is a User-agent in robots.txt?▼

User-agent specifies which crawler the following rules apply to. Use * for all crawlers. Use specific names like Googlebot, Bingbot or GPTBot for individual crawlers. You can have multiple User-agent sections with different rules. Googlebot respects the * section if no specific Googlebot section exists.

Where should I put the robots.txt file?▼

The robots.txt file must be at the root of your domain: https://yourdomain.com/robots.txt. It cannot be in a subdirectory. Each subdomain needs its own robots.txt file. The file must return a 200 HTTP status code and be publicly accessible without authentication.

What is the Sitemap directive in robots.txt?▼

The Sitemap directive tells search engines the location of your XML sitemap file. Add the full URL: Sitemap: https://example.com/sitemap.xml. Supported by Google, Bing and other major search engines. Adding your sitemap in robots.txt helps crawlers discover all your pages efficiently.

What is Crawl-delay in robots.txt?▼

Crawl-delay suggests how many seconds a crawler should wait between requests. Googlebot ignores Crawl-delay entirely - to control Googlebot robots txt crawl rate, use the Crawl Rate settings in Google Search Console instead. Some other crawlers like Bingbot do respect it. Use sparingly - overly aggressive crawl-delay values can slow search engine discovery of your content.

What robots.txt should I use for WordPress?▼

WordPress robots.txt should block /wp-admin/ (with Allow for admin-ajax.php), /wp-login.php, search results (/?s=), trackbacks and feeds. Always allow /wp-content/ so Google can crawl theme CSS, plugin JavaScript and images. Load the WordPress preset in the generator above for a ready-to-use configuration.

Related tools