Category: Uncategorized

Three files commonly sit at a site root and shape how crawlers discover and access content. They are frequently discussed together and differ enormously in status.

robots.txt

Standardised as RFC 9309. It states which paths a crawler may fetch. It is advisory: compliant crawlers honour it, and nothing enforces it.

This has a consequence for measurement that is easy to miss. If a crawler is disallowed and you observe no traffic from it, you have learned nothing about whether it would otherwise have visited. Absence caused by your own configuration is not a finding about the crawler.

sitemap.xml

A well-established convention for listing a site’s URLs, optionally with a <lastmod> modification date. Crawlers use it as a discovery aid, not as an instruction.

The <lastmod> value is a claim the site makes about itself. Crawlers are known to discount it when it proves unreliable — sites that stamp every URL with today’s date teach crawlers to ignore the field entirely.

A useful self-check: compare each sitemap <lastmod> against the Last-Modified header the same URL actually returns. They are two independent declarations of the same fact. When they disagree, at least one is wrong, and a crawler can observe the same inconsistency.

llms.txt

A proposed convention for a Markdown file giving LLM-based agents a curated map of a site. It is not ratified by any standards body, no crawler is obliged to fetch it, and no major operator has publicly committed to honouring it.

It costs little to publish and may prove useful. What can be stated honestly today is a count: how many times anything actually requested the file, and which User-Agents did. On most sites that number is currently very low or zero, and zero is a legitimate result rather than a misconfiguration.

What none of them do

None of these files causes indexing. They assist discovery and state access preferences. Publishing them correctly removes obstacles; it does not create demand for your content.

These are methodological notes on interpreting crawler traffic in origin server logs. They are deliberately cautious, because this data is easy to over-read.

A CDN makes traffic disappear

When a CDN serves a response from its cache, the origin server is never contacted and no log line is written. Origin logs therefore undercount crawler activity by an unknown amount that depends entirely on your caching configuration.

Comparing two crawlers is only meaningful if both were subject to identical caching. If HTML is cached at the edge, the comparison is not valid.

Absence of conditional requests is not a fault

Nothing in HTTP obliges a crawler to revalidate. A crawler that never sends If-None-Match is using the protocol correctly; it has simply chosen not to use that mechanism. The correct reading is “this crawler did not revalidate during the observation window”, not “this crawler is misbehaving”.

A 304 is a statement about bytes

It says the representation matching the client’s validator has not changed. It says nothing about what the client does next, whether any downstream system re-processed the content, or whether anything at all was understood. What happens after the response leaves the server is entirely invisible.

Correlation is not causation

If a crawl follows a URL submission, that is a sequence of events, not a demonstrated effect. Crawlers visit on their own schedule for their own reasons. Establishing that a submission caused a crawl requires a control group of comparable URLs that were not submitted, observed over the same window.

Behaviour changes over time

Crawl frequency and fetch behaviour change as operators update their infrastructure, without announcement. A result measured in one month may not reproduce in the next. Always record the observation window alongside any finding — a conclusion without dates attached cannot be reproduced or checked.

IndexNow is an open protocol that lets a site notify participating search engines that a URL has changed. It is supported by Bing, Yandex, Seznam and Naver. It is simple, and it is routinely over-interpreted.

How it works

You generate a key and publish it as a text file at your site root. You then POST a list of changed URLs along with that key. The engine fetches your key file to confirm you control the host, and queues the URLs for consideration.

There is no account and no secret. The key is public by design; it proves host control and nothing else.

Response codes

• 200 — submitted successfully
• 202 — accepted, key validation still pending
• 400 — invalid format
• 403 — key not valid, usually a missing or mismatched key file
• 422 — the URLs do not belong to the declared host
• 429 — rate limited

A 403 after a period of working normally usually means the key file stopped being served — a redeploy removed it, or a CDN began serving a cached 404 for that path.

Three different events

These are constantly conflated, and keeping them separate is the whole discipline:

• Submission — the engine received your request. This is what a 200 confirms.
• Crawl — the engine fetched the URL. Observable in your access log.
• Indexing — the engine stored it and may serve it in results. Observable only through that engine’s own tools, and only partially.

A successful submission does not promise a crawl. A crawl does not promise indexing. Any claim that submission “gets you indexed” is skipping two steps that are not in your control.

Measuring it honestly

You can measure the interval between a submission and the next crawl of that URL. That interval is a timeline, not a demonstrated effect: crawlers also visit on their own schedule. Establishing that the submission caused the crawl requires a control group of comparable URLs that were not submitted, observed over the same window.

A web crawler discovers URLs, decides when to fetch them, retrieves them over HTTP, and passes the result to whatever system it serves.

Discovery

Crawlers find URLs through links on pages they have already fetched, XML sitemaps, direct submissions through protocols such as IndexNow, and redirects. A page with no inbound links and no sitemap entry may simply never be found.

Scheduling

Crawlers do not fetch everything constantly. They estimate how often a page changes and schedule accordingly, balanced against a crawl budget that limits load on any one host. A page that has never changed may be revisited rarely; one that changes daily may be revisited daily.

Fetching

Behaviour at fetch time varies considerably between operators. Some send conditional requests to avoid re-downloading unchanged pages. Some always fetch unconditionally. Some execute JavaScript; many do not. None of this is mandated by any specification.

Identity

Crawlers identify themselves with a User-Agent string. That string is a self-declared header and can be set to anything at all, which scrapers routinely exploit to avoid being blocked.

The documented way to verify a crawler is forward-confirmed reverse DNS: resolve the IP to a hostname, check the hostname belongs to the operator, then resolve that hostname back and confirm it returns the original IP. The final step is what stops someone who controls their own PTR record.

Not every operator publishes reverse DNS records. Several publish IP ranges instead, and some publish nothing, in which case identity cannot be verified by this method at all.

robots.txt

The Robots Exclusion Protocol, standardised as RFC 9309, lets a site state which paths a crawler may fetch. It is advisory. Compliant crawlers honour it; nothing enforces it.

This has a consequence for measurement: if a crawler is disallowed and you observe no traffic from it, you have learned nothing about whether it would otherwise have visited.

The default Nginx “combined” log format has been standard since the 1990s. It is adequate for counting page views and useless for studying caching.

What combined records

Client IP, timestamp, request line, status code, bytes sent, referrer and User-Agent. That is all.

What it cannot record

It has no field for the If-None-Match or If-Modified-Since headers the client sent, and none for the ETag or Last-Modified the server returned. So a combined log can tell you a request returned 304, but never why, nor which validator earned it.

Why JSON rather than more fields

You could bolt extra quoted fields onto the combined format. That breaks the first time a User-Agent contains a double quote — and User-Agent is attacker-controlled, so it will eventually contain whatever someone wants. A JSON format with proper escaping is unambiguous to parse.

The variables that matter

• $http_if_none_match — the validator the client presented
• $http_if_modified_since — the date-based equivalent
• $sent_http_etag — the validator the server actually returned
• $sent_http_last_modified — the modification time claimed
• $sent_http_cache_control — the caching policy advertised
• $request_time — total handling time
• $http_cf_connecting_ip — the real client IP when a CDN is in front

The $sent_http_* family reads the response headers, which is how a server records what it actually sent rather than what it intended to send. That distinction catches real bugs.

The limitation no log format fixes

When a CDN serves a response from its edge cache, the origin is never contacted and no line is written. Origin logs undercount by an unknown amount that depends entirely on caching configuration. No choice of log format changes that, and any analysis built on origin logs has to state it.

A conditional request is an ordinary request carrying a precondition. The server evaluates it and either fulfils the request normally or responds that nothing has changed.

The first request

The client asks for a resource and the server responds with the content plus validators:

HTTP/1.1 200 OK
ETag: "9f2c1a7b3e4d5f6a"
Last-Modified: Wed, 12 Feb 2025 09:14:22 GMT
Cache-Control: public, max-age=600, must-revalidate

The second request

Later, the client asks again, this time presenting the validator it holds:

GET /test-page/ HTTP/1.1
If-None-Match: "9f2c1a7b3e4d5f6a"

If the resource is unchanged, the server replies:

HTTP/1.1 304 Not Modified
ETag: "9f2c1a7b3e4d5f6a"

No body. The client reuses what it already has.

Testing it properly

Confirming a 304 arrives is only half the test. You also need to confirm the server is actually comparing rather than answering 304 unconditionally.

• Send the correct ETag — expect 304
• Send a deliberately wrong ETag — expect 200
• Send the correct Last-Modified — expect 304
• Send a very old date — expect 200

A server that returns 304 for all four is broken in a way that would silently invalidate any measurement built on top of it.

A common failure

Plenty of sites emit an ETag and then ignore If-None-Match entirely, returning a full 200 with a fresh body every time. The header is present, so the site looks correctly configured, but no client ever benefits from it. Test rather than assume.

HTTP defines two ways for a server to identify a version of a resource. They are not interchangeable, and a site can easily end up in a state where they contradict each other.

Last-Modified

A timestamp indicating when the resource last changed. The client sends it back in If-Modified-Since, and the server compares dates.

Its weakness is resolution: HTTP dates have one-second granularity. Two changes within the same second are indistinguishable. It also requires the server to genuinely know when the content changed, which is harder than it sounds for a page assembled from many sources.

ETag

An opaque identifier for a specific version of a representation. The client sends it back in If-None-Match, and the server compares strings.

ETags have no resolution limit and can be derived directly from content, so they detect any change. The cost is that computing a content-derived ETag usually means generating the response before you can decide whether to send it.

Strong and weak validators

A strong ETag ("abc123") means byte-for-byte identical. A weak one (W/"abc123") means semantically equivalent — useful when compression or trivial formatting differences should not count as a change.

This matters in practice: when a server gzips a response, it may convert a strong ETag to a weak one, so the value the client sees differs from the value the application generated.

When the two disagree

Consider editing only a page’s meta description. The modification timestamp updates, so Last-Modified changes. But if that metadata does not appear in the rendered output, the body is byte-identical and the ETag does not change.

A client using If-None-Match now gets 304. A client using If-Modified-Since gets 200. Both are behaving correctly according to the specification. The site is simply making two different claims about whether it changed.

When both validators are present, If-None-Match takes precedence — RFC 9110 is explicit about this.

HTTP caching exists to avoid transferring data that the client already has. It operates on two distinct mechanisms, and confusing them is the source of most caching bugs.

Freshness

A response carries a lifetime, usually via Cache-Control: max-age. While the response is within that lifetime it is considered fresh, and a cache may reuse it without contacting the origin server at all. No request is made, so the origin sees nothing.

Validation

Once a response goes stale, the cache does not simply discard it. It asks the origin whether the copy it holds is still good, using a validator it received earlier. If the origin says nothing has changed, the cache reuses its stored copy.

This is where 304 Not Modified comes in. The origin answers with headers only, no body, and the client reuses what it already had. The request still happens — it is just very small.

Why the distinction matters for measurement

A fresh cached response produces no origin request and no log line. A validated response produces a request that appears in the log with status 304. If you are counting crawler activity from origin logs, freshness makes traffic disappear while validation makes it cheap but visible.

Cache-Control directives worth knowing

• max-age=600 — fresh for ten minutes
• must-revalidate — once stale, the cache must check before reusing
• no-cache — may be stored, but must be revalidated every time
• no-store — must not be stored at all
• public / private — whether shared caches may store it

Note that no-cache does not mean “do not cache”. That is no-store. The naming is unfortunate and the confusion is common.