toolbox.app
Zero Data Retention

robots.txt Generator

With plain-English notes.

Write a valid robots.txt in seconds: one clean rule group, your sitemap, the paths crawlers should skip, and an optional block for AI training crawlers, each explained in plain English.

Zero retention: your data never leaves this browser. Verify it yourself in DevTools

Loading the toolbox.

About 2robots

How Can You Verify That Nothing Leaves This Browser?

You do not have to take our word that your data stays here. Developer tools show every request a page makes, so you can watch this page work and confirm that nothing is sent. The check takes under a minute.

  1. Open your browser developer tools with F12, or Cmd+Option+I on a Mac, and select the Network tab.
  2. Reload the page and let it finish. You will see the page's own files, its web fonts, and one anonymous page-view request. Nothing has been typed yet, so none of them can contain your data.
  3. Click the clear icon in the Network tab so that the log is empty.
  4. Use the tool with your own input.
  5. Look at the Network tab again. It is still empty. For extra proof, turn on offline mode after the page loads and the tool keeps working.

Only two things can add a line to that log, and neither carries your data. Submitting the feedback form sends the vote and comment you typed. Typing characters in a script the page has not displayed yet, such as Cyrillic or Greek, can make the browser fetch the matching font file, and that request names only the font.

The production site also sends a Content-Security-Policy header that tells your browser to refuse connections to anywhere except this site and the anonymous analytics endpoint, so a coding mistake could not quietly send your data elsewhere.

What Does robots.txt Do?

robots.txt is a plain text file at the root of a host, such as https://example.com/robots.txt, that tells crawlers which paths they may fetch. It is defined in RFC 9309. Rules are grouped under User-agent lines, Disallow and Allow lines are prefix matches, and the most specific matching rule wins.

It is a request, not a lock. Well-behaved crawlers such as Googlebot and Bingbot follow it. Others may not, and anyone can read the file, so never list a secret path in it.

Does Disallow Remove a Page From Google?

Not reliably. A disallowed URL can still be indexed from links pointing to it, shown without a description. To keep a page out of search results, let it be crawled and send a noindex robots meta tag or X-Robots-Tag header, or put it behind authentication. Blocking and noindex together are worse than either: if the crawler cannot fetch the page, it never sees the noindex.

What Changed From the Original Tool?

The vanilla 2robots wrote a separate User-agent: * group for every checkbox and allowed "allow all" and "disallow all" to be on at once. Crawlers handle repeated groups differently, so the result was ambiguous. This version always writes one coherent group, validates every path and the sitemap URL, and explains each line.

Should You Block AI Crawlers?

That is a business decision. The optional AI group lists the published user agents for AI training and answer crawlers, including GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider, Applebot-Extended, and meta-externalagent. Google-Extended and Applebot-Extended only control AI use; they do not affect normal search indexing. Compliance is voluntary and the list changes, so review it from time to time.

What Other Tools Are in the Generators Lane?

  • 2uuid: UUID v4 and v7, ULID, and NanoID in bulk, plus a decoder that reads timestamps back out.
  • 2gitignore: Pick your languages, frameworks, OS, and editor; get one merged, de-duplicated .gitignore.
  • 2csp: Build a Content-Security-Policy header or meta tag, with nonces, hashes, strict-dynamic, and Report-Only.
  • 2htaccess: Apache rules for HTTPS, www canonical, compression, caching, headers, file blocking, and SPA fallback.

Elsewhere in the toolbox: 2hash, 2base, 2otp, 2xml, and more on the 2toolbox home page. Part of the 2suite.io privacy-first ecosystem.

Frequently Asked Questions

Where does robots.txt go?

At the root of each host, such as https://example.com/robots.txt. Each subdomain needs its own.

Does Google honor Crawl-delay?

No. Google ignores it. Bing and Yandex honor it.

Can I have more than one Sitemap line?

Yes. List as many as you need; Sitemap lines apply to every crawler.

What does an empty Disallow mean?

Allow everything. It is the same as having no robots.txt, without the 404s in your logs.

Was this helpful?