Malilion Tools

robots.txt Generator and Tester

Generate a robots.txt and test, with the same rules, whether a given path ends up allowed or blocked — plus one-click blocking for the common AI crawlers.

Runs in your browser: input and output are never sent to the server, and the platform does not store your input.

Input

Not stored

Supports the * wildcard and a trailing $. These are added to the scenario's own rules; pick “Custom” to use only your own.

When both match, the longer rule wins; on a tie, Allow wins.

Shows which rule allows or blocks this path.

Google ignores this directive; Bing and Yandex honour it.

Output

Fill in the fields on the left, then run.

How to use

  1. 1Pick a scenario, then add your own Disallow and Allow paths if you need them.
  2. 2Fill in the sitemap URL and use “Test a path” to confirm important pages are not blocked by accident.
  3. 3Save the result as robots.txt at the site root (https://your-domain/robots.txt).

FAQ

Can robots.txt hide sensitive pages?

No. It is a request to well-behaved crawlers, and the file itself is publicly readable — so it publishes a list of the paths you wanted hidden. Protect anything real with authentication or permissions.

How is Disallow different from noindex?

Disallow means do not fetch; noindex means do not list. A disallowed page is never read, so its noindex is never seen, and it can still be indexed without a snippet because of inbound links. To keep a page out of results, let crawlers read it and put noindex on the page.

Why should I not block CSS and JS?

Search engines render pages to judge layout and mobile-friendliness. Block the styles and scripts and the crawler sees a broken page, which scores worse than the page you were hiding.

Does blocking AI crawlers actually work?

It works for crawlers that publish their user agent and honour robots.txt — GPTBot, ClaudeBot, Google-Extended and the like. Crawlers that ignore the file will not stop because of it.

Is my input uploaded?

No. Both the generation and the path test run in your browser; no sign-in and no quota.