Use the procedure on a representative sample, keep the evidence, and record the next action.
Check access without blocking useful pages
- Fetch robots.txt for each host and protocol. Record user-agent groups, allow/disallow rules, sitemap declarations, and the change history.
- List important canonical URLs and test whether crawling is allowed. Check that robots.txt is not being used to remove a page from the index.
- Check login, staging, search, and filter URLs separately. Use robots.txt only to manage crawling of confirmed duplicate or low-value URL spaces; protect private pages with authentication.
- Deploy a small change, fetch robots.txt again, test representative URLs in Search Console, and keep a rollback copy.