Three files commonly sit at a site root and shape how crawlers discover and access content. They are frequently discussed together and differ enormously in status.
robots.txt
Standardised as RFC 9309. It states which paths a crawler may fetch. It is advisory: compliant crawlers honour it, and nothing enforces it.
This has a consequence for measurement that is easy to miss. If a crawler is disallowed and you observe no traffic from it, you have learned nothing about whether it would otherwise have visited. Absence caused by your own configuration is not a finding about the crawler.
sitemap.xml
A well-established convention for listing a site’s URLs, optionally with a <lastmod> modification date. Crawlers use it as a discovery aid, not as an instruction.
The <lastmod> value is a claim the site makes about itself. Crawlers are known to discount it when it proves unreliable — sites that stamp every URL with today’s date teach crawlers to ignore the field entirely.
A useful self-check: compare each sitemap <lastmod> against the Last-Modified header the same URL actually returns. They are two independent declarations of the same fact. When they disagree, at least one is wrong, and a crawler can observe the same inconsistency.
llms.txt
A proposed convention for a Markdown file giving LLM-based agents a curated map of a site. It is not ratified by any standards body, no crawler is obliged to fetch it, and no major operator has publicly committed to honouring it.
It costs little to publish and may prove useful. What can be stated honestly today is a count: how many times anything actually requested the file, and which User-Agents did. On most sites that number is currently very low or zero, and zero is a legitimate result rather than a misconfiguration.
What none of them do
None of these files causes indexing. They assist discovery and state access preferences. Publishing them correctly removes obstacles; it does not create demand for your content.