Util Tools What is robots.txt?
robots.txt is a plain text file at the root of your site that tells crawlers which parts they may
or may not request. It uses simple rules: a User-agent line names the crawler, then
Disallow and Allow lines list paths. This generator builds a valid file with
per-crawler rules, your sitemap, and one-click blocking of AI crawlers.
Where it goes and why you need one
The file must live at yourdomain.com/robots.txt; crawlers look for it there and nowhere else. You
do not strictly need one (a site with no robots.txt is fully crawlable), but it is useful for keeping crawlers
out of admin areas, search-result pages, and duplicate content, and for pointing them to your sitemap. Including
a sitemap line helps search engines discover your pages.
Allow vs. Disallow, and what it really controls
Disallow blocks a path from being crawled; Allow creates an exception within a blocked section. An important caveat: robots.txt is a crawling instruction, not a security or privacy control. It politely asks compliant crawlers to stay out, but it does not prevent access, and a disallowed URL can still be indexed if linked elsewhere. To truly keep a page out of search, use a noindex tag or authentication.
Blocking AI crawlers
A growing reason to edit robots.txt is to control whether AI companies can scrape your content to train models. You can disallow specific user-agents such as GPTBot, Google-Extended, CCBot, and others. This tool includes presets so you can block the major AI crawlers in one click if you prefer your content not be used for training, while keeping normal search engines welcome.
Last updated: July 2026