These are methodological notes on interpreting crawler traffic in origin server logs. They are deliberately cautious, because this data is easy to over-read.
A CDN makes traffic disappear
When a CDN serves a response from its cache, the origin server is never contacted and no log line is written. Origin logs therefore undercount crawler activity by an unknown amount that depends entirely on your caching configuration.
Comparing two crawlers is only meaningful if both were subject to identical caching. If HTML is cached at the edge, the comparison is not valid.
Absence of conditional requests is not a fault
Nothing in HTTP obliges a crawler to revalidate. A crawler that never sends If-None-Match is using the protocol correctly; it has simply chosen not to use that mechanism. The correct reading is “this crawler did not revalidate during the observation window”, not “this crawler is misbehaving”.
A 304 is a statement about bytes
It says the representation matching the client’s validator has not changed. It says nothing about what the client does next, whether any downstream system re-processed the content, or whether anything at all was understood. What happens after the response leaves the server is entirely invisible.
Correlation is not causation
If a crawl follows a URL submission, that is a sequence of events, not a demonstrated effect. Crawlers visit on their own schedule for their own reasons. Establishing that a submission caused a crawl requires a control group of comparable URLs that were not submitted, observed over the same window.
Behaviour changes over time
Crawl frequency and fetch behaviour change as operators update their infrastructure, without announcement. A result measured in one month may not reproduce in the next. Always record the observation window alongside any finding — a conclusion without dates attached cannot be reproduced or checked.