Use this when a crawl spike, repeated 5xx responses, or an unexpected bot load appears. A User-Agent string is a claim, not proof of identity.
Build a useful log sample
- Export timestamp, source IP, User-Agent, method, path, status, response bytes, and request duration. Keep a fixed window and timezone.
- Group by claimed crawler, status, and path. Count requests and distinct IPs separately; do not infer people or visits from raw requests.
Verify identity before changing access
- For Googlebot, verify the source IP with Google’s published IP ranges or the documented reverse-DNS procedure. A matching User-Agent alone is insufficient.
- Compare verified requests with robots rules, crawl rate, and server errors. Escalate only the exact abusive IP/path pattern; keep evidence and a rollback.
Example and expected result
- Example: a log row says
GET /page 200 "Googlebot". For a real IP, run reverse DNS, confirm an official Google crawler hostname, then forward-resolve it and verify the original IP returns. Record status, URL, response code and time. The sample IP 203.0.113.24 is documentation-only. - If the two DNS checks do not match, mark the request unverified; do not decide from User-Agent alone. This procedure covers Google only; use each other provider’s published verification method. Restrict log access and keep IP data only for the configured retention period.