what is crawling?
walking a site by following links to discover urls, as distinct from extracting data out of a page you already have.
a crawler keeps a frontier of urls to visit, fetches them, extracts links, deduplicates, and repeats until it hits a limit. scraping is what you do with each page once you have it; crawling is how you find pages worth fetching. the hard parts are discovery and control. url canonicalisation matters because trailing slashes, tracking parameters and session ids all produce duplicates of the same page. traps like infinite calendars and faceted search generate unlimited urls that lead nowhere. scope needs explicit rules on depth, domain and path. politeness means one connection and a fixed delay per host, plus robots.txt and any crawl-delay it declares. where a sitemap exists it is almost always faster and kinder than link discovery. because a crawl hits many pages on one site, it meets per ip rate limits sooner than a targeted scrape, so a rotating pool plus a queue that paces per host both matter.