Indexing (SEO)

Indexing is the process by which a search engine analyzes a crawled page and stores it in its index, the database it draws from to answer search queries. A page that is not indexed cannot appear in search results, no matter how well it is optimized.

What is indexing?

Search engines operate in three steps: crawling (discovering and downloading a page), indexing (analyzing and storing that page's content, structure and signals), and ranking (deciding where an indexed page appears for a given query). Indexing sits in the middle: a page can be crawled without ever being indexed, if the search engine judges it low-value, duplicate, or explicitly told not to index it. Only indexed pages compete for rankings.

How indexing works

After crawling a URL, a search engine renders the page, extracts its text, links and structured data, then decides whether to add it to the index. Site owners control indexing directives through a meta tag or an HTTP header:

<meta name="robots" content="index, follow">

The equivalent HTTP header, useful for non-HTML files such as PDFs, sets the same instruction server-side:

X-Robots-Tag: index, follow

Setting either to noindex instead removes the page from consideration, even if it was previously indexed and even if it is still crawled.

Indexing states in Google Search Console

• Indexed: the page is in the index and eligible to rank.
• Crawled, currently not indexed: Google visited the page but chose not to index it, often a content quality or duplication signal.
• Discovered, currently not indexed: Google knows the URL exists but has not crawled it yet, usually a crawl budget or priority issue.
• Excluded by 'noindex' tag: the page explicitly told search engines not to index it.
• Duplicate, submitted URL not selected as canonical: Google indexed a different version of a near-identical page.

Indexing vs crawling vs ranking

Best practices and common pitfalls

The most common indexing mistake is an accidental noindex tag left over from a staging environment, which silently removes a page from search results with no error message. A thin or duplicate page rarely gets indexed at all, since search engines allocate crawl and index budget toward content that adds distinct value. Submitting a sitemap and linking internally from already-indexed pages both speed up discovery of new URLs, but neither guarantees indexing if the content itself does not clear the quality bar.

Checking and improving index status

Google Search Console's URL Inspection tool reports a page's exact indexing status and the reason if it is excluded, and a "Request indexing" action can prompt a faster recrawl after a fix. A site:example.com search gives a rough, non-exhaustive estimate of indexed pages. Pages worth indexing benefit from being reachable within a few clicks of the homepage, included in the XML sitemap, and free of conflicting signals such as a canonical tag pointing elsewhere.

Indexing at BeBranded

We audit indexing status as a first step of every SEO & GEO engagement, since a technically perfect page that is not indexed generates zero organic traffic regardless of its content quality.