This repository contains educational example scrapers for popular web scraping targets using the ScrapFly web scraping API and Python.
Most Scrapers use a simple web scraping stack:
- Python version 3.10+
- Scrapfly's Python SDK for sending HTTP request, bypass blocking and parsing the HTML using the built-in parsel selector.
- asyncio for writing concurrent code using the async/await syntax.
- JMESPath and nested-lookup for JSON parsing when needed.
- loguru for logging.
To learn more about web scraping see our full tutorials on how to scrape these targets (and many others) see the scrapeguide directory.
For AI agents and LLM tools, llms.txt lists every scraper with its functions and sample output, and explains how to run a scraper and how to write a new one.
Below is the list of available web scrapers for the supported domains along with their scrape guide, sample datasets, and status. 👇
The status comes from the nightly test run, which also orders the table: scrapers whose tests pass come first, failing and untested ones last.
