Create robots.txt files for your website. Control how search engines and AI crawlers access your site with Allow, Disallow rules, and sitemap references.
Place robots.txt in your website root directory (e.g., example.com/robots.txt)
| Directive | Description | Example |
|---|---|---|
| User-agent | Specifies which crawler the rules apply to | User-agent: Googlebot |
| Disallow | Blocks access to a path or pattern | Disallow: /admin/ |
| Allow | Explicitly permits access to a path | Allow: /api/public |
| Sitemap | Points to XML sitemap location | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Seconds between requests | Crawl-delay: 10 |
| Host | Preferred domain (Yandex) | Host: https://www.example.com |
User-agent: * Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /admin/ Disallow: /api/ Disallow: /*?* User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / Sitemap: https://example.com/sitemap.xml
Visual Rule Builder
Create complex robots.txt rules with an intuitive interface. No syntax knowledge required.
Multiple User Agents
Support for Googlebot, Bingbot, GPTBot, and 15+ other crawlers with predefined options.
Quick Templates
Start with pre-built templates for common scenarios: Allow All, Block All, Block AI Bots, E-commerce.
Sitemap Integration
Add multiple sitemap URLs to help search engines discover your content more efficiently.
AI Bot Blocking
Easily block AI crawlers like GPTBot, ChatGPT-User, CCBot, and Anthropic bots from your site.
Advanced Options
Set crawl-delay directives and host preferences for fine-grained crawler control.
Control which pages search engines can index. Block admin areas, duplicate content, and internal pages.
Prevent AI companies from using your content for training by blocking GPTBot, CCBot, and similar crawlers.
Block cart, checkout, and account pages from indexing while allowing product pages to be crawled.
Block all crawlers from staging or development environments to prevent them from appearing in search results.
Love our free tools? Discover our AI-powered products that help businesses automate customer communication.
Automate phone calls with AI. Handle appointments, inquiries, and support 24/7 in 40+ languages.
Build WhatsApp chatbots without coding. Automate support and boost sales with 98% open rates.
Add an intelligent chatbot to your website. Convert visitors, answer questions, and capture leads 24/7.
Translate and dub videos in 100+ languages with natural AI voices. Reach global audiences in minutes.
Turn every call into insights with AI transcription, sentiment analysis, and quality scoring for support and sales teams.
Visual deal pipeline with drag-and-drop kanban, Gmail integration, and contact management. Close more deals faster.
Connect customers and staff on a masked number so personal numbers stay private. Click-to-call masking with a simple REST API.
Local, toll-free, and virtual numbers across India from Rs 349/month. Call forwarding, IVR, and free number porting included.
Elastic SIP trunking at 50 paisa/min. Connect any PBX or AI voice agent to the Indian PSTN. No contracts, self-serve setup.
Recurring revenue for Shopify. Pause/skip, build-a-box, prepaid plans, and a customer portal with UPI Autopay & Razorpay.