Sitemap URL Extractor

Enter a domain or sitemap URL. Sitemaps are discovered via robots.txt and common paths, nested indexes and .gz files are followed, and every URL comes back with lastmod, changefreq and priority. Monitor mode returns only URLs that are new since your last run.

Features

• Auto-discovery, nested indexes, .xml.gz
• Regex and date filters
• Monitor mode: only new URLs
• Built for RAG ingestion and content tracking

Example: Get notified of a competitor's new blog posts

{
  "startUrls": [
    "competitor.com"
  ],
  "includePatterns": [
    "/blog/"
  ],
  "onlyNewSinceLastRun": true
}

Only URLs published since the previous run, with lastmod. Schedule daily.