Design a Web Crawler

Requirements

A web crawler (also called a spider or bot) starts from some known pages, downloads them, extracts the links inside and repeats. Search engines use it to collect pages to index. Other uses are price monitoring, archiving and training-data collection.

Functional:

• Start from a list of seed URLs (the starting pages) and discover new pages through links.
• Download pages and store their content for later processing (indexing, analysis).
• Revisit pages over time to pick up changes.

Non-functional:

• Polite: never overload a website, and obey its rules (robots.txt, explained below).
• Scalable: hundreds of pages per second, spread across many machines.
• Robust: the web is full of broken HTML, endless redirects, slow servers and traps. One bad page must not stop a worker.
• Extensible: new content types (images, PDFs) should be easy to add.