Reading Nginx Access Logs

Written by

in

The default Nginx “combined” log format has been standard since the 1990s. It is adequate for counting page views and useless for studying caching.

What combined records

Client IP, timestamp, request line, status code, bytes sent, referrer and User-Agent. That is all.

What it cannot record

It has no field for the If-None-Match or If-Modified-Since headers the client sent, and none for the ETag or Last-Modified the server returned. So a combined log can tell you a request returned 304, but never why, nor which validator earned it.

Why JSON rather than more fields

You could bolt extra quoted fields onto the combined format. That breaks the first time a User-Agent contains a double quote — and User-Agent is attacker-controlled, so it will eventually contain whatever someone wants. A JSON format with proper escaping is unambiguous to parse.

The variables that matter

  • $http_if_none_match — the validator the client presented
  • $http_if_modified_since — the date-based equivalent
  • $sent_http_etag — the validator the server actually returned
  • $sent_http_last_modified — the modification time claimed
  • $sent_http_cache_control — the caching policy advertised
  • $request_time — total handling time
  • $http_cf_connecting_ip — the real client IP when a CDN is in front

The $sent_http_* family reads the response headers, which is how a server records what it actually sent rather than what it intended to send. That distinction catches real bugs.

The limitation no log format fixes

When a CDN serves a response from its edge cache, the origin is never contacted and no line is written. Origin logs undercount by an unknown amount that depends entirely on caching configuration. No choice of log format changes that, and any analysis built on origin logs has to state it.