Distributed scraping system
A high-throughput crawling and ingestion system for retail and automotive data across the US and Canada.
- Python
- Django
- Scrapy
- Playwright
- Zyte
- Oxylabs
- Redis
- MongoDB
- AWS
What I built
I led the development of a distributed scraping system that cut end-to-end processing time from 4.5 hours to under an hour, about 75%, and lowered infrastructure cost by about 40%.
How it works
- 200+ concurrent workers process 1.4M+ products daily.
- Enterprise proxy networks (Zyte, Oxylabs, ISP and mobile proxies) are combined with region-aware routing for geo-accurate requests across the US and Canada.
- Adaptive concurrency and queue-based load balancing keep the system stable under peak load.
- Automotive crawlers extract VIN-based vehicle data: pricing, dealer metadata, listing history and market trends.
- Retail crawlers extract product catalogs, pricing, variations, availability and structured metadata.
- Third-party sources are integrated through Zyte’s scraping API, SERP APIs and product data APIs.
- Scraped data is normalized into consistent schemas for market analysis, price comparison and feed ingestion.