From raw log lines to answerable questions
The web server writes one structured record per request. On its own that is an append-only text file that grows by tens of thousands of lines a day and answers nothing. The analytics layer turns it into something you can query.
The pipeline
Each stage does one job, and each stage can be inspected independently, which matters when a number looks wrong and you need to find out where it came from.
- Collection — Nginx writes one JSON object per request, including the conditional-request headers the client sent and the freshness headers the server returned.
- Parsing — a Python parser reads new bytes only, tracking a byte offset per file so ingestion is resumable and log rotation cannot cause loss or duplication.
- Classification — each request is labelled as a known crawler, an unknown bot, or a browser, using a maintained catalogue of documented User-Agent strings.
- Storage — rows land in SQLite, deduplicated by a hash of the raw log line.
- Presentation — a dashboard queries that database. It never touches the web server.
What is stored per request
Timestamp, client IP, HTTP method, requested URL, response status, bytes sent, referrer, User-Agent, request duration, the If-None-Match and If-Modified-Since headers the client sent, and the ETag, Last-Modified and Cache-Control headers the server returned.
Storing both sides of the exchange is the point. Knowing that a request returned 304 is only meaningful alongside knowing which validator the client presented to earn it.
Data quality rules applied
- Timestamps are stored in UTC, so string ordering is chronological ordering.
- Every row records which log file it came from, so a bad import can be traced and removed.
- Rows are deduplicated by line hash, so re-reading a file cannot inflate a count.
- Test traffic and real traffic are never mixed. The database is reset before a measurement period begins.
Questions the data can answer
Which crawlers reached the origin and how often. Which pages they requested. What status codes they received. Which crawlers send conditional requests, and how often those produce a 304. How long after a content change a crawler returned.
Questions it cannot answer
Anything about what happens after the response leaves the server. Whether content was read, stored, understood or used. Whether any of it affects search ranking or how an AI system represents the site. Those are outside what an access log can observe, and no amount of analysis will extract them from one.