Separate discovery, crawling, indexing, and serving

Explanation

Search has several stages. For example, Google describes URL discovery, crawling, processing and indexing, followed by serving results. A successful server response by itself does not prove that a URL was found or added to the index; inclusion in the index does not guarantee that the page will appear for a particular query. First define which state you are checking.

Fictional example

https://learn.example.test/robots returns 200 OK, but the initial HTML contains a “Loading…” heading and an internal link to the lesson. The main text is added after an API request, which in this training scenario returns 403 without cookies. This observation suggests a possible issue with making the main content available during rendering. It does not prove that a search engine will never see the page or that it is not indexed.

Practice

Recreate the training page /robots on your own staging environment or local mockup. Separately record the HTTP response, whether an internal link exists, the result of a separate robots.txt check, whether the main content is available, and the Search Console result only if you have a verified property with an accessible URL. Do not submit learn.example.test to public tools; mark unavailable checks “not checked.”

Checklist

• The exact URL and final URL after redirects were checked.
• The HTTP status was recorded, not just the browser view.
• robots.txt and noindex directives were checked.
• The main text and links are available in the HTML/rendered DOM.
• Unknowns are explicitly recorded.

Common mistake

“The URL returns 200, so it is in Google.” A 200 confirms a successful HTTP response for that check; it does not confirm indexing or appearance in results.

Primary sources

• How Google Search works
• Google Search technical requirements
• URL Inspection in Search Console