Home/Tools/Sitemap Crawler
OPEN SOURCEFREE · PYTHON · MIT LICENSE

Sitemap Crawler: Find Recently Updated Pages on Any Website

A free, open-source Python script that reads a site's sitemap and tells you exactly which pages changed in the last 24 hours - yours, or a competitor's. No SEO tool subscription required.

View on GitHub See Technical SEO Service
What it does

Your sitemap already knows what changed.
Most people never check.

Discovers the sitemap itself
Reads robots.txt for declared sitemap URLs, falling back to common paths like sitemap.xml if none are listed.
Handles nested sitemap indexes
Large sites split their sitemap into an index of smaller files. The crawler follows that structure recursively until it reaches every actual page URL.
Filters by last-modified time
Every URL comes with its lastmod timestamp. The script keeps only what changed inside the time window you set - 24 hours by default.
Why we built it

Three things it's actually good for.

Publishing cadence checks

Confirm your own site is actually publishing or updating on the schedule you think it is, instead of assuming your CMS or team is keeping pace.

Competitor monitoring

See when a competitor refreshes a page without manually re-checking it every week - useful before a content gap analysis or a technical SEO review.

No subscription needed

Everything it reads is public - robots.txt and sitemap XML. No API key, no paid SEO tool required.

How to use it

It's a script, not a web app.

This one's for anyone comfortable running Python from a terminal - developers, technical SEOs, or anyone who'd rather own the code than pay for a dashboard. Clone it, install two dependencies, and run it.

# clone and install

git clone https://github.com/jinnatulhasan/sitemapcrawler.git

cd sitemapcrawler

pip install -r requirements.txt

# edit the websites list in sitemapcrawler.py, then run

python sitemapcrawler.py

Full setup, usage, and how the code works is documented in the README on GitHub.

FAQ

Questions you're
probably asking.

Do I need to know Python to use this?
You need enough comfort with a terminal to clone a repo, install two pip packages, and edit a list of URLs in a text file. There's no web interface - it's a script you run yourself.
Does this work on any website?
It works on any site with a public sitemap that includes lastmod dates - which is most reasonably well-built sites. If a site doesn't publish lastmod data, there's nothing to filter by.
Is this free, and can I modify it?
Yes - it's MIT licensed. Use it, change it, ship it inside your own tooling. Attribution is appreciated but not required.
Can Whizz People run this for me instead?
Content freshness checks like this are part of what we look at in a technical SEO audit - if you'd rather not run scripts yourself, that's exactly the kind of thing our Technical SEO service covers.
Get started

Want us to handle
the technical SEO too?

Content freshness is one small piece of a proper technical audit. Talk to us about the rest.

Book a Call View on GitHub