# Reliable website deduplication with per-page content hashing

> WebsiteReader now computes a unique content hash per crawled URL, fixing skip_if_exists for multi-page crawls.

- Published: 2026-01-28
- Author: Agno Team
- Category: Changelog
- Canonical: https://agno-com-nine.vercel.app/articles/reliable-website-deduplication-with-per-page-content-hashing
- Markdown: https://agno-com-nine.vercel.app/articles/reliable-website-deduplication-with-per-page-content-hashing.md

WebsiteReader now computes a unique content hash per crawled URL, fixing skip_if_exists for multi-page crawls. This ensures accurate per-page deduplication, reduces redundant ingestion, and saves processing cost during re-crawls.

Details:

- Correct per-page deduplication for predictable skip_if_exists behavior
- Fewer unnecessary writes and tokens when re-indexing multi-page sites
- Action required: Clear existing website crawl entries in your knowledge store before re-indexing to avoid duplicates

Who this is for: Teams maintaining search indexes, documentation portals, or knowledge bases sourced from websites.