robots.txt Generator
Pick a preset or block a folder. A valid robots.txt appears as you type.
Your own rules. Editing any field switches the chip back to Custom.
One folder or pattern per line. A missing leading slash is added for you.
Use this to reopen one path inside a folder you blocked. More specific rules win.
A full address, including https://. Bare hosts are refused.
Seconds. Google ignores it.
Block a folder, allow everything, or start from a preset. The file builds as you type and never leaves this tab.
01 Capability
What it does
Small details that make the job quicker.
- Writes a real file, not a template: each user-agent gets its own group, and the allow and block paths appear in the order you typed them.
- Presets cover allow all, block all and the usual WordPress folders; editing any field switches the chip back to Custom.
- A path typed without a slash is corrected to /path, and duplicate rules are dropped before they reach the output.
- The sitemap field refuses a bare host or a missing scheme instead of writing a line crawlers cannot follow.
- Crawl-delay is optional and checked as whole seconds; leave it blank and no line is written.
- Copy takes the whole file and Download saves it as robots.txt — plain text, no comments, no tracking, no upload.
02 Questions
Questions
Short answers, no small print.
Does robots.txt keep a page out of Google?
No. A Disallow rule stops crawlers reading the page; it does not remove a page that is already indexed. Google can still list a blocked URL if other sites link to it. To keep a page out of the index, let it be crawled and send a noindex tag, or remove it and request removal in Search Console.
Should I block everything while a site is in development?
Disallow: / is the right short-term lock. Remember that it also blocks Google from seeing a noindex tag, so do not rely on both at once. Remove the rule when the site opens, and check the live file at your domain before launch, because a cached or forgotten block is the classic launch-day mistake.
Which sitemap address should I use?
A full https:// address, usually https://example.com/sitemap.xml. This tool writes one Sitemap line because one is enough for most sites. If your CMS generates several sitemaps, copy the block and add another Sitemap line by hand; the format is the same.
Do I still need Allow rules if Disallow: / is set?
Only for exceptions. Crawlers use the most specific matching rule, so Allow: /wp-admin/admin-ajax.php can reopen one file inside a blocked /wp-admin/ folder. The WordPress preset does exactly that, because blocking admin-ajax breaks the dashboard for logged-in editors.
Is Crawl-delay respected?
Google ignores it. Some other crawlers, including Bing and Yandex, may honour it. Leave it blank unless your host or the crawler's documentation asks for one, and fix the server or cache before asking bots to slow down.
Where does the file go, and is anything uploaded?
Put robots.txt at the root of the domain, so it answers at https://example.com/robots.txt, and serve it as text/plain. Paths, user-agent names and the sitemap URL are turned into text in this tab; Astra does not upload, log or store them, and the download is built locally.