SEO & Performance

Page not indexed by Google? A developer's diagnostic checklist

Separate discovery, crawl access, indexing directives and Google's selected canonical when diagnosing a page missing from search. Know what local checks can prove.

A page can load in your browser and still be absent from Google’s index. It may be undiscovered, blocked from crawling, excluded by a directive or treated as a duplicate. Those are different problems.

Start with the exact public URL and gather evidence before changing SEO settings. A localhost preview cannot establish the state of the deployed page or Google’s index.

1. Inspect Google’s known state

Use URL Inspection in Search Console for a property you control. Distinguish information about Google’s indexed version from a live test of the page’s current accessibility.

Record the reported state, inspection date and canonical information. Do not treat an older crawl result as evidence that a newly deployed fix has already been processed.

A broad search using site: can provide a clue, but it is not an exhaustive index report. Use Search Console for the investigation rather than judging the whole site by one search result page.

2. Check the actual public response

Request the final HTTPS URL without login cookies. For an illustrative URL:

curl -sS -D /tmp/page-headers.txt \
  -o /tmp/page-body.html \
  'https://example.com/guides/example/'

Inspect the saved status and headers as well as the HTML. This command does not follow redirects, so a redirect response tells you to inspect its destination separately. A header-only request may not behave exactly like a normal page request.

Google’s technical requirements include crawl access, a successful response and indexable content. Meeting those requirements makes a page eligible; it does not guarantee indexing.

3. Separate robots.txt from noindex

Check the applicable robots.txt rules and the page’s robots directives, including any X-Robots-Tag response header. A proxy or CDN can add a header that is absent from your application template.

An intentional noindex instruction might look like:

<meta name="robots" content="noindex">

Google must be able to crawl a page to see its noindex instruction. Blocking the URL in robots.txt while expecting Google to read that instruction creates conflicting behavior. See Google’s noindex documentation.

Do not remove exclusions from private or utility pages just to increase an indexed-page count. Decide which URLs should appear in search first.

4. Inspect canonical intent

Check the declared canonical URL and compare it with Google’s selected canonical where Search Console provides that information. A page may be a valid alternate version of another page rather than a broken indexing case.

Look for accidental template behavior: every article pointing to the homepage, a production canonical pointing at staging, or conflicting declarations in the HTML and headers.

If the intended page moved, review canonical URLs versus redirects instead of applying both indiscriminately.

5. Review discovery and content

Link to the page from an appropriate crawlable page and include the intended canonical URL in the sitemap. A sitemap helps discovery; it does not order Google to index every listed URL.

Inspect whether the public response contains the main content. If JavaScript supplies it later, examine the rendered output as well as the initial HTML. Check for an error message masquerading as a successful page.

Ask whether the page provides a distinct, useful answer. Producing more nearly identical URLs is not a reliable way to increase useful search coverage.

6. Turn the findings into one targeted fix

Finding Appropriate next step
Wrong redirect destination Correct the URL mapping and test the destination
Accidental noindex Remove the directive where appropriate and verify the deployed response
Incorrect canonical Correct the preferred URL and related signals
Page inaccessible without login Decide whether the content is intended to be public
No clear technical blocker Review Search Console evidence, duplication and content usefulness

After deployment, repeat the relevant technical checks and use the available Search Console submission process where appropriate. Allow time for recrawling; repeatedly requesting the same URL is not a substitute for resolving the cause.

For WordPress output checks, see the SEO metadata tutorial. For slow pages, use the separate performance diagnosis guide rather than assuming speed alone explains an indexing decision.