User-agent: * Disallow: /admin/ Disallow: /private/ Allow: /public/ Sitemap: https://example.com/sitemap.xml
How it works
The robots.txt file sits at your site root (yourdomain.com/robots.txt) and tells search engine crawlers which URLs they may and may not access. It's the first file Googlebot reads when it discovers your domain. A misconfigured robots.txt can accidentally block your entire site from being indexed — or waste crawl budget on irrelevant pages. The Robots.txt Generator produces a correct, deployment-ready robots.txt from a guided form.
Key directives explained: - **User-agent**: specifies which crawler the following rules apply to. `User-agent: *` applies to all crawlers. `User-agent: Googlebot` applies only to Google's crawler. - **Disallow**: paths the crawler must not access. `Disallow: /admin/` blocks all URLs under /admin/. `Disallow: /` blocks the entire site — the most dangerous misconfiguration. - **Allow**: overrides a Disallow for specific paths. Used when you want to block a directory but allow specific files within it. - **Sitemap**: declares the location of your XML sitemap — tells crawlers where to find all your pages. - **Crawl-delay**: requests a pause between requests (in seconds). Supported by Bing but ignored by Googlebot — use Google Search Console's crawl rate settings instead for Google.
Common paths to disallow: - `/admin/`, `/wp-admin/` — backend admin panels - `/checkout/`, `/cart/`, `/account/` — transactional pages with no SEO value - `/search?` — internal search result pages (faceted navigation creates crawl budget waste) - `/cdn-cgi/` — Cloudflare internal paths - `/staging/`, `/dev/` — development directories
Critical warning: robots.txt Disallow does NOT remove pages from the index. Pages already indexed remain indexed until Google re-crawls them and finds either the Disallow rule or a noindex meta tag. To delist a page, use noindex or Google Search Console's URL removal tool.
How to use: 1. Select the crawlers to configure (All / Googlebot only / Bingbot only / Custom). 2. Add Disallow paths one per line. 3. Enter your sitemap URL. 4. Download robots.txt and deploy to your site root.
Privacy: all generation runs in the browser.
Frequently Asked Questions
- Your entire site becomes uncrawlable by all compliant search engine bots. Googlebot will stop crawling new pages and will eventually remove indexed pages from search results as they can no longer be verified. This is one of the most damaging SEO mistakes possible — a single character error in robots.txt can delist an entire site. Always test changes at yourdomain.com/robots.txt immediately after deploying and verify in Google Search Console's robots.txt tester.
- No. Robots.txt prevents crawling — it does not prevent indexing. Google can index a URL it has never crawled if that URL appears in a sitemap or is linked to from other indexed pages. To prevent a page from appearing in search results, use <meta name='robots' content='noindex'> in the page's HTML head, or return an X-Robots-Tag: noindex HTTP header. Disallowing in robots.txt alone does not remove existing indexed URLs.
- For WordPress sites, yes. The /wp-admin/ directory contains backend administration pages that have no value to search engines. Disallowing it reduces crawl budget consumption on non-public pages. However, keep /wp-admin/admin-ajax.php accessible — some WordPress plugins and theme functions rely on AJAX calls that go through this endpoint, and blocking it can break front-end features even though the URL itself isn't a public page.
- The Crawl-delay directive requests a minimum pause (in seconds) between crawler requests. Bing, Yandex, and most other crawlers respect it. Google explicitly ignores Crawl-delay in robots.txt. To adjust Googlebot's crawl rate, use Google Search Console: Settings → Crawl rate. For most sites, default Googlebot crawl rate is appropriate — only request slower crawl rates if your server is struggling under Googlebot's load.