About this project
This is a proof of concept for measuring web crawler behaviour. It is a real, self-hosted WordPress site running on a cloud virtual machine behind Cloudflare, instrumented so that every request reaching the origin server is recorded in detail.
Why a real site
Crawler behaviour cannot be observed in a laboratory. Search and AI crawlers visit real domains on their own schedule, and their behaviour depends on the site actually existing, being reachable, and serving content that changes over time. A synthetic test harness would tell you nothing about how a real crawler responds.
The technical stack
- Ubuntu LTS on a small cloud virtual machine
- Nginx configured with a JSON access log that records conditional-request headers
- PHP-FPM and WordPress for editable content
- MariaDB as the content store — post modification times come from here
- Cloudflare in front, for TLS and CDN behaviour
- Python and SQLite for log parsing and analysis
What makes this measurable
WordPress does not send ETag or Last-Modified headers on front-end pages by default, so there is normally nothing for a crawler to revalidate against. A must-use plugin adds both, derived from the actual rendered page and the genuine post_modified_gmt timestamp, and honours conditional requests properly.
Because the values are content-derived rather than fabricated, editing a page produces a real, observable change in both validators. That is what makes the experiments repeatable.