web-crawler

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf

Python scraper based on AI

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Ingest, parse, and optimize any data format ➡️ from documents to multimedia ➡️ for enhanced compatibility with GenAI frameworks

The Ultimate Information Gathering Toolkit

简单易用的Python爬虫框架，QQ交流群：597510560

Scalable Python web scraping scripts for +40 popular domains

Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!

扫描“微信读书”已购图书并下载本地PDF的爬虫

Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation.

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

Undetected web-scraping & seamless HTML parsing in Python!

The only open-source toolkit that can download SEC EDGAR financial reports and extract textual data from specific item sections into nice & clean structured JSON files. Presented at WWW 2025 @ Sydney, Australia (https://dl.acm.org/doi/10.1145/3701716.3715289)

Opensource Korean chatbot framework

A crawl workstation: View, Control, and Crawl. Solr CrawlDB, Tika, Vue 3.

The simple, easy to use command line web crawler.

Data Analysis & Mining for lagou.com

旨在将网易云、酷狗、QQ、酷我等各音乐平台集于一体

Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns! (ISML=international saimoe; 2022 ISML is last ISML)

A simple distributed crawler for zhihu && data analysis