Paresh Pawar Guide – How crawler activity is measured
Every request that reaches the origin server is written to a structured access log. A Python parser reads that log, classifies each request, and stores it in a SQLite database. A dashboard then reports on the result.
What the data can show
- Which crawlers reached the origin, how often, and over what period
- Which pages each crawler requested
- The distribution of HTTP response codes per crawler
- Which crawlers send conditional requests, and how often they receive a 304
- The interval between a content change and the next crawler visit
- Whether anything requests
robots.txt,sitemap.xmlorllms.txt
What the data cannot show
This is the important part. The measurements above describe HTTP traffic and nothing more.
- A
304response means the bytes did not change. It does not mean any AI system re-read, understood or re-evaluated the content. - Requests served from a CDN cache never reach the origin, so they are absent from these counts entirely.
- A User-Agent string is self-declared and can be forged, so crawler identity is a claim unless independently verified.
- No conclusion about search ranking or AI visibility can be drawn from any of this.
Crawler categories
Crawlers are grouped as search engines, AI training crawlers, AI assistants performing live retrieval, SEO tools, social link unfurlers, and monitoring services. The grouping is descriptive, based on each operator’s own published documentation.