Robots.txt

What robots.txt is

Robots.txt is a plain text file placed at the root of a website that tells search engine crawlers which parts of the site they are allowed to access. When a well behaved crawler like Googlebot visits, it checks this file first and follows its instructions before fetching anything else. It is one of the oldest and simplest tools for managing how automated visitors interact with a site.

The file lives at a fixed location, yoursite.com/robots.txt, and uses a small, standardised syntax. It is publicly visible to anyone, which is an important point: robots.txt guides crawlers, it does not hide or secure anything.

Despite its age, robots.txt remains one of the first things a crawler looks for, which gives a very small file a lot of influence. It is part of a long standing agreement between site owners and well behaved bots, a voluntary protocol rather than an enforced barrier. That voluntary nature is central to understanding both what it is good at and where it stops, because a crawler that chooses to ignore it faces no technical obstacle.

What robots.txt is for

Its main purpose is crawl management. Large sites have pages that add no value in search, such as internal search results, admin areas, filtered duplicate URLs or staging content. Telling crawlers not to spend time on these helps them focus their limited crawl budget on the pages that matter, which is especially useful on big sites.

It is also used to point crawlers to the sitemap, and to keep bots away from sections that would waste resources. Used well, it makes crawling more efficient and keeps low value URLs out of a crawler's path.

Crawl budget is the concept that makes robots.txt genuinely useful on larger sites. Search engines allocate a finite amount of crawling to each site, and every request spent on a useless URL is a request not spent on a page that matters. By steering bots away from endless filtered variations, internal search pages and other low value paths, you help them reach and refresh your important content faster, which can have a real effect on how quickly changes are picked up.

How robots.txt works

The file is a list of rules grouped by which crawler they apply to. The core directives are simple.

• User-agent: names the crawler a rule applies to, with an asterisk meaning all crawlers.
• Disallow: tells the crawler not to access a path.
• Allow: makes an exception within a disallowed area.
• Sitemap: points to the location of the site's sitemap.

For example, a rule can block all crawlers from an admin folder while allowing everything else. The syntax is forgiving but literal, so a small mistake in a path can have a big effect.

A crucial misunderstanding

The most common and dangerous mistake is treating robots.txt as a way to hide pages or keep them out of search. It does neither. Blocking a page in robots.txt only stops compliant crawlers from fetching it; the URL can still appear in search results if other pages link to it, just without a description. And because the file is public, listing a secret path actually advertises it.

To keep a page out of search results, you use a noindex directive on the page instead, which requires the crawler to be able to access it. To protect private content, you use real authentication. Robots.txt is a traffic guide for bots, not a lock.

The confusion between blocking and hiding causes some of the most damaging mistakes in SEO. Because a blocked page can still be indexed as a bare URL when other sites link to it, and because the crawler can no longer read a noindex tag it is forbidden from fetching, the two tools can even work against each other. The reliable rule is to let a crawler read a page you want removed, so it can see the noindex, and to reserve robots.txt for steering rather than suppression.

Robots.txt risks and best practices

Because a single line can block an entire site, robots.txt deserves care. A stray disallow rule left over from a staging environment can quietly remove a site from search, one of the most damaging and easily missed SEO errors. The best practice is to keep the file minimal, test it, and never block resources like CSS or JavaScript that engines need to render the page.

Regular checks matter. Search consoles offer tools to test the file and see how crawlers interpret it, so verifying after any change is a sensible habit, especially before and after a site launch.

Robots.txt at BeBranded

At BeBranded we treat robots.txt as a small file with outsized consequences, and we review it on every Webflow project we ship. We make sure a staging site's blanket disallow never survives into production, that the sitemap is referenced, and that nothing engines need to render the page is blocked. We combine it correctly with noindex directives for genuine deindexing, since the two tools do different jobs. The result for clients is a site that crawlers can access cleanly and efficiently, with no accidental blocks quietly costing them visibility.

We document the intent behind each rule too, so a line added for a good reason is not mistaken later for one that can be removed. A robots.txt without context invites accidental changes, and on a live site those changes are easy to make and slow to notice. Keeping it minimal, commented and reviewed at launch is a small habit that prevents some of the most avoidable losses in search visibility.