crawling

Scrapling

+1k

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

scrapy/scrapy

+140

Scrapy, a fast high-level web crawling & scraping framework for Python.

getmaxun/maxun

+73

Turn any website into a structured API. Extract, automate, search and monitor the web.

apify/crawlee

+58

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

exa-mcp-server

+39

Exa MCP for web search and web crawling!

donsetch

+33

Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.

ai.robots.txt

+21

A list of AI agents and robots to block.

crawlee-python

+17

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

holiday-cn

+16

📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告