A web crawler discovers URLs, decides when to fetch them, retrieves them over HTTP, and passes the result to whatever system it serves.
Discovery
Crawlers find URLs through links on pages they have already fetched, XML sitemaps, direct submissions through protocols such as IndexNow, and redirects. A page with no inbound links and no sitemap entry may simply never be found.
Scheduling
Crawlers do not fetch everything constantly. They estimate how often a page changes and schedule accordingly, balanced against a crawl budget that limits load on any one host. A page that has never changed may be revisited rarely; one that changes daily may be revisited daily.
Fetching
Behaviour at fetch time varies considerably between operators. Some send conditional requests to avoid re-downloading unchanged pages. Some always fetch unconditionally. Some execute JavaScript; many do not. None of this is mandated by any specification.
Identity
Crawlers identify themselves with a User-Agent string. That string is a self-declared header and can be set to anything at all, which scrapers routinely exploit to avoid being blocked.
The documented way to verify a crawler is forward-confirmed reverse DNS: resolve the IP to a hostname, check the hostname belongs to the operator, then resolve that hostname back and confirm it returns the original IP. The final step is what stops someone who controls their own PTR record.
Not every operator publishes reverse DNS records. Several publish IP ranges instead, and some publish nothing, in which case identity cannot be verified by this method at all.
robots.txt
The Robots Exclusion Protocol, standardised as RFC 9309, lets a site state which paths a crawler may fetch. It is advisory. Compliant crawlers honour it; nothing enforces it.
This has a consequence for measurement: if a crawler is disallowed and you observe no traffic from it, you have learned nothing about whether it would otherwise have visited.