# AI Crawler POC Test Site

This file follows the llms.txt convention (https://llmstxt.org). It lists
the main pages of this site with their real last-modified dates.

## Pages

- [About](https://paresh-pawar-agency.online/about/): About this project This is a proof of concept for measuring web crawler behaviour. It is a real, self-hosted WordPress site running on a cloud virtual machine b... (last modified 2026-08-15)
- [AI Analytics](https://paresh-pawar-agency.online/ai-analytics/): Paresh Pawar Guide - How crawler activity is measured Every request that reaches the origin server is written to a structured access log. A Python parser reads... (last modified 2026-08-15)
- [Blog](https://paresh-pawar-agency.online/blog/): Articles on HTTP caching, conditional requests and crawler behaviour. (last modified 2026-08-15)
- [Contact](https://paresh-pawar-agency.online/contact/): Get in touch This site is a technical proof of concept. If you are a crawler operator and something here is misconfigured from your perspective, or you have que... (last modified 2026-08-15)
- [Data Analytics](https://paresh-pawar-agency.online/data-analytics/): From raw log lines to answerable questions The web server writes one structured record per request. On its own that is an append-only text file that grows by te... (last modified 2026-08-15)
- [Home](https://paresh-pawar-agency.online/): This site is a controlled test environment for observing how web crawlers behave when pages provide HTTP freshness and revalidation information. What is being m... (last modified 2026-08-15)
- [Services](https://paresh-pawar-agency.online/services/): Technical capabilities demonstrated HTTP freshness and revalidation Correct generation of ETag and Last-Modified validators for dynamically generated pages, and... (last modified 2026-08-15)
- [Test Page](https://paresh-pawar-agency.online/test-page/): This page is the controlled surface for the HTTP freshness experiments. It is deliberately plain: no widgets, no comment form, no dynamic elements of any kind.... (last modified 2026-08-15)

## Articles

- [Sitemaps, robots.txt and llms.txt](https://paresh-pawar-agency.online/sitemaps-robots-and-llms-txt/): Three files at the site root, with very different levels of standardisation and enforcement. (last modified 2026-08-14)
- [Notes on Crawler Behaviour](https://paresh-pawar-agency.online/notes-on-crawler-behaviour/): What origin access logs can and cannot tell you about crawler activity. (last modified 2026-08-14)
- [What IndexNow Does and Does Not Do](https://paresh-pawar-agency.online/what-indexnow-does-and-does-not-do/): A submission protocol, its response codes, and the three distinct events people routinely conflate. (last modified 2026-08-13)
- [How Web Crawlers Work](https://paresh-pawar-agency.online/how-web-crawlers-work/): Discovery, scheduling, fetching, and why crawler identity is harder to establish than it looks. (last modified 2026-08-13)
- [Reading Nginx Access Logs](https://paresh-pawar-agency.online/reading-nginx-access-logs/): Why the default log format is not enough for caching analysis, and what to record instead. (last modified 2026-08-12)
- [Conditional Requests Explained](https://paresh-pawar-agency.online/conditional-requests-explained/): What a conditional request looks like on the wire, and how to test that your server honours one. (last modified 2026-08-12)
- [ETag vs Last-Modified](https://paresh-pawar-agency.online/etag-vs-last-modified/): Two validators, two failure modes, and why a site can end up disagreeing with itself. (last modified 2026-08-11)
- [Understanding HTTP Caching](https://paresh-pawar-agency.online/understanding-http-caching/): How HTTP caching actually works, and the difference between freshness and validation. (last modified 2026-08-10)

## Resources

- [Sitemap](https://paresh-pawar-agency.online/wp-sitemap.xml): full URL list in XML with lastmod
- [robots.txt](https://paresh-pawar-agency.online/robots.txt): crawler access policy