Robots.txt Configuration
🤖 AI & LLM Bots
💡 Allowing AI bots enables your content to be used for training models and appear in AI responses
Helps search engines find all your pages
Delay between each bot request (0 = no delay)
Paths that bots should not explore
Useful for allowing subfolders within a blocked folder
⚙️ Custom Rules
Generated robots.txt File
# Robots.txt generated by BeBranded
# Date: ${new Date().toLocaleDateString('en-US')}
User-agent: *
Allow: /
⚠️ Place this file at the root of your site: yoursite.com/robots.txt
📝 In Webflow: Project Settings → SEO → Robots.txt → Paste the generated code