Explanation
robots.txt controls crawler access to URLs, but it is not a reliable way to remove a URL from results: the address may be known from elsewhere. A noindex directive expresses that an accessible page should not be included in search. A crawler generally needs to fetch the page to read noindex. Check actual server responses and access controls, not just the text file.
Fictional example
The training staging configuration for /robots has both Disallow and noindex. A crawler that is disallowed may not fetch the HTML and therefore may not read the meta tag. First, the team records the intended state of the page and checks what each directive does; it then changes and retests only the staging configuration. Nothing changes in production. Private and test sections need a separate decision by the owner.
Practice
On your own staging environment or local mockup, check /robots: robots.txt; the meta robots tag in HTML; the X-Robots-Tag response header; status and redirects; and any WAF/CDN restrictions or login requirements. Compare each directive with the intended state. Keep any rule changes in staging; production requires the owner, review, and a rollback plan.
Checklist
robots.txtwas checked for the correct host and protocol.- Response headers were checked, not just the HTML.
Disallowandnoindexare treated as different directives.- Private content is protected with authentication, not only
robots.txt. - Results are recorded separately for each relevant search crawler.
Common mistake
Adding Disallow and assuming the page has been removed from the index. If a public page should not be indexed, use an appropriate indexing directive and do not block the crawler from reading it; confidential data requires access control.