Technical capabilities demonstrated
HTTP freshness and revalidation
Correct generation of ETag and Last-Modified validators for dynamically generated pages, and correct handling of If-None-Match and If-Modified-Since conditional requests with 304 Not Modified responses.
Crawler identification and verification
Classification of incoming requests into known crawlers, unknown bots and normal browsers, based on a maintained catalogue of documented User-Agent strings. Crawler claims are verified where possible using forward-confirmed reverse DNS, because a User-Agent header alone proves nothing about who is calling.
Access log analysis
A structured JSON access log capturing timestamp, client IP, method, URI, status, response size, referrer, User-Agent, request duration, the conditional headers the client sent, and the freshness headers the server returned.
Site file management
Explicit robots.txt policy with a named group for each documented AI crawler, XML sitemaps carrying real lastmod values, and an llms.txt file generated from actual published content.
Index monitoring
URL submission through the IndexNow protocol, with the literal HTTP status of each submission recorded, and index status retrieved from the Bing Webmaster Tools API where that data is available.