Skip to main content
WorkUtilities

robots.txt Generator

Build a valid robots.txt with user-agent rules, allow/disallow paths, and sitemap URL. Copy or download instantly.

Runs in your browserFree ยท No signup
Disallow
Allow

Live Preview

User-agent: *

About robots.txt Generator

Our free robots.txt Generator builds a valid robots.txt file with user-agent rules, allow/disallow paths, crawl-delay, and sitemap URL โ€” with live preview.

One-click presets for WordPress, Next.js/Vercel, allow-all, block-all, and AI crawler blocking (GPTBot, CCBot).

Copy to clipboard or download as robots.txt instantly. No signup, runs entirely in your browser.

Read the full guide: robots.txt generator guide โ€” syntax & validation

When Should You Use robots.txt Generator?

New website launch

Create a robots.txt before going live.

Block admin areas

Prevent crawlers from indexing /admin/ or /wp-admin/.

Block AI crawlers

Add GPTBot, CCBot, and Meta AI blocking rules.

Next.js / Vercel

Apply standard _next and /api/ disallow patterns.

Sitemap declaration

Add your sitemap.xml URL for search engines.

Frequently Asked Questions

What is a robots.txt file?

A text file at your domain root that tells crawlers which pages to crawl, following the Robots Exclusion Protocol.

Does robots.txt block pages from search results?

No โ€” it prevents crawling but not indexing if other sites link to the page. Use noindex for that.

How do I block GPTBot?

Add User-agent: GPTBot and Disallow: / to your robots.txt. Reputable AI companies honor this.

Can I allow some pages while blocking others?

Yes โ€” Allow rules override Disallow when both match. More specific rules take precedence.

Is this generator free?

Yes. WorkUtilities robots.txt Generator is free with no signup.

Help Us Improve

Your feedback helps us build better tools.