Beeknoee Logo
Đăng nhậpĐăng ký miễn phí

monosans/proxy-scraper-checker

Public

Fast proxy scraper and checker written in Rust. Collects HTTP, SOCKS4 and SOCKS5 proxies from any number of sources, verifies each one really works, and writes JSON and plain text with response time, exit IP, ASN and geolocation. Single binary for Windows, Linux, macOS and Android, with an interactive TUI.

asyncclifree-proxygeolocationhttp-proxyproxyproxy-checkerproxy-grabberproxy-listproxy-scrapeproxy-scraperproxyscrapeproxyscraperrustsocks4socks4asocks5socks5htokiotui
Star1.3kFork280Watch21Mở trên GitHub
  • Stars1.3k+1 tuần này
  • Forks280
  • Watchers21
  • Issues0
  • Ngày tạo30/04/2021
  • Cập nhật gần nhất31/10/2026
  • Ngôn ngữRust
  • Giấy phépMIT
README.md35.7 KBRaw

proxy-scraper-checker

TUI Demo

A proxy scraper and checker written in Rust.

It collects HTTP/SOCKS4/SOCKS5 proxies from any number of sources, verifies each one really works, and writes the survivors out with response times, geolocation and network ownership attached.

Features

  • Async Rust, shipped as a single binary for Windows, Linux, macOS and Android. Nothing to install.
  • Proxies are pattern-matched out of raw text, HTML or JSON, so any source works untouched. scheme://user:pass@host:port and CIDR ranges are understood, from URLs or local files.
  • Every proxy has to fetch a real URL in full to survive, so ones that connect and then stall get dropped. Results are deduplicated across sources.
  • Response time, exit IP, ASN and city-level geolocation come from offline MaxMind databases, so there are no per-proxy API calls.
  • The interactive TUI shows per-protocol progress and logs as the run goes.
  • Output is JSON with metadata, plus ready-to-paste plain text lists.

Related

If you want proxies without running anything, monosans/proxy-list is updated regularly using this tool.

Safety warning

Checking opens hundreds of simultaneous connections to untrusted hosts. Your ISP may read that as abusive traffic, and cheap routers can drop connectivity when their NAT table fills. Consider a VPN, and lower max_concurrent_checks if a run destabilizes your network.

Quick start

Every option is documented inline in config.toml. The binary reads config.toml from the current directory; set PROXY_SCRAPER_CHECKER_CONFIG to point it somewhere else.

Pre-built archives are produced by CI from the latest commit on main and published as the latest release, see Verifying downloads. Open the section for your system.

Windows

  1. Download the archive for your processor:

    Processor Download
    64-bit Intel or AMD (almost every PC) Download
    ARM64 (Snapdragon and other ARM laptops) Download
    32-bit Download

    Not sure? Open Settings → System → About and look at "System type": x64-based processor means 64-bit Intel or AMD, ARM-based processor means ARM64, and 32-bit operating system needs the 32-bit build whatever the processor.

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run proxy-scraper-checker.exe. If SmartScreen shows "Windows protected your PC", click "More info", then "Run anyway".

macOS

  1. Download the archive for your processor:

    Processor Download
    Apple Silicon (M1 and newer) Download
    Intel Download

    Not sure? Apple menu → About This Mac shows "Chip: Apple M…" on Apple Silicon and "Processor: … Intel …" on Intel.

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from Terminal in that folder:

    ./proxy-scraper-checker

    If macOS refuses because it cannot verify the developer, clear the download quarantine and run it again:

    xattr -d com.apple.quarantine proxy-scraper-checker
Linux

  1. Run uname -m and download the archive from the matching row. The glibc build suits most distributions (Ubuntu, Debian, Fedora, Arch and so on). Take the musl build on Alpine, or if the glibc build refuses to start.

    Processor uname -m glibc musl
    64-bit Intel or AMD x86_64 Download Download
    64-bit ARM (Raspberry Pi 3 and newer on a 64-bit OS, ARM servers) aarch64 Download Download
    32-bit ARMv7 (Raspberry Pi 2 and newer on a 32-bit OS) armv7l Download Download
    32-bit ARMv6 (Raspberry Pi 1, Zero, Zero W) armv6l Download Download
    32-bit Intel or AMD i686 Download Download
    64-bit RISC-V riscv64 Download Download
    LoongArch loongarch64 Download Download
    64-bit PowerPC, little-endian ppc64le Download —
    64-bit PowerPC, big-endian ppc64 Download —
    32-bit PowerPC ppc Download —
    IBM Z s390x Download —
    64-bit SPARC sparc64 Download —
    32-bit ARMv5 armv5tel Download Download
    32-bit x86 without SSE2 (Pentium III, Athlon XP and older) i686 Download Download
    32-bit ARMv7 without hardware floating point armv7l Download Download
    32-bit ARMv6 without hardware floating point armv6l Download Download
    32-bit ARMv7 with NEON, Thumb-2 build armv7l Download —

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from that folder:

    ./proxy-scraper-checker
Android / Termux

Install Termux from F-Droid, not Google Play (why?).

  1. Install with one command:

    bash <(curl -fsSL 'https://raw.githubusercontent.com/monosans/proxy-scraper-checker/main/termux.sh')

  2. Edit the config in a text editor:

    nano ~/proxy-scraper-checker/config.toml

  3. Run it:

    cd ~/proxy-scraper-checker && ./proxy-scraper-checker

The script picks the right archive by itself. To fetch one by hand, match the output of getprop ro.product.cpu.abi:

getprop ro.product.cpu.abi Download
arm64-v8a Download
armeabi-v7a Download (without NEON)
armeabi Download
x86_64 Download
x86 Download
FreeBSD

  1. Run uname -m and download the archive from the matching row:

    Processor uname -m Download
    64-bit amd64 Download
    32-bit i386 Download

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from that folder:

    ./proxy-scraper-checker
Docker on Windows

The Docker image logs to stdout instead of showing the interactive TUI.

  1. Install Docker Desktop.

  2. Download the archive for your processor:

    Processor Download
    64-bit Intel or AMD (almost every PC) Download
    ARM64 (Snapdragon and other ARM laptops) Download

    Not sure? Open Settings → System → About and look at "System type": x64-based processor means 64-bit Intel or AMD, ARM-based processor means ARM64.

  3. Extract it to a folder and edit config.toml.

  4. Build and run from that folder:

    docker compose build
docker compose up --no-log-prefix --remove-orphans

Results land in out inside that folder; output.path is ignored in Docker.

Docker on Linux / macOS

The Docker image logs to stdout instead of showing the interactive TUI.

  1. Install Docker Compose.

  2. Run uname -m and download the archive from the matching row:

    Processor uname -m Download
    64-bit Intel or AMD (including Intel Macs) x86_64 Download
    64-bit ARM (Apple Silicon Macs, Raspberry Pi on a 64-bit OS) aarch64, arm64 on macOS Download
    32-bit ARMv7 armv7l Download
    32-bit Intel or AMD i686 Download
    64-bit PowerPC, little-endian ppc64le Download
    64-bit RISC-V riscv64 Download
    IBM Z s390x Download

    The ppc64le, riscv64 and s390x archives compile the program during docker compose build, so the first build takes a while.

  3. Extract it to a folder and edit config.toml.

  4. Build and run from that folder. The build arguments make the files in ./out belong to your user:

    docker compose build --build-arg UID=$(id -u) --build-arg GID=$(id -g)
docker compose up --no-log-prefix --remove-orphans

Results land in ./out; output.path is ignored in Docker.

Build from source

  1. Install the Rust toolchain, see https://rust-lang.org/tools/install/

  2. Clone the repository:

    git clone https://github.com/monosans/proxy-scraper-checker.git
cd proxy-scraper-checker

  3. Build a release binary with the TUI enabled:

    cargo build --features tui --release --locked

  4. Run with the TUI:

    cargo run --features tui --release --locked

The binary lands in target/release/proxy-scraper-checker on Linux/macOS, or target\release\proxy-scraper-checker.exe on Windows.

Verifying downloads

Every archive is built by GitHub Actions from the public source and published by the CI workflow straight from GitHub's runners, never from someone's computer:

  • Each release links to a build provenance attestation that GitHub signs through Sigstore. It lists the SHA-256 of every archive together with the commit and the workflow run that built it, and only that workflow run could have produced the signature. Whatever went into a build, the source and the workflow included, is therefore in the public history of this repository.
  • Releases are immutable: once published, their files cannot be replaced.

To check a downloaded archive without installing anything:

  1. Compute its SHA-256: Get-FileHash <archive> in PowerShell, shasum -a 256 <archive> on macOS, sha256sum <archive> on Linux and Termux, sha256 <archive> on FreeBSD.
  2. Open the "Build provenance" link in the notes of the latest release and find the same hash next to the archive name. The page also shows the commit and the workflow run it was built from. Only the latest release is kept; attestations of older builds stay on the attestations page.

To check the signature itself, use the GitHub CLI:

gh attestation verify <archive> \
  --repo monosans/proxy-scraper-checker \
  --signer-workflow monosans/proxy-scraper-checker/.github/workflows/ci.yml

The Docker images are attested the same way, and the prebuilt Docker archives pin the image by digest, so docker compose build uses exactly the image CI built.

Output

out/
├── proxies.json          working proxies with metadata, compact
├── proxies_pretty.json   the same data, indented
└── proxies/
    ├── all.txt           every proxy, with its protocol prefix
    ├── http.txt          one file per enabled protocol, no prefix
    ├── socks4.txt
    └── socks5.txt

A text line is host:port, or username:password@host:port for a proxy that needs credentials, with protocol:// in front of either in all.txt.

{
  "protocol": "socks5",
  "username": null,
  "password": null,
  "host": "1.2.3.4",
  "port": 1080,
  "timeout": 0.42,
  "exit_ip": "5.6.7.8",
  "asn": {
    "autonomous_system_number": 12345,
    "autonomous_system_organization": "Example Networks"
  },
  "geolocation": {
    "country": { "iso_code": "US", "names": { "en": "United States" } },
    "city": { "names": { "en": "Chicago" } },
    "location": {
      "latitude": 41.85,
      "longitude": -87.65,
      "time_zone": "America/Chicago"
    }
  }
}

timeout is how long the whole request took, in seconds: connection, request and the complete response. That is also what sort_by_speed orders by. exit_ip is the address the target site actually saw, so comparing it with host spots proxies that forward through somewhere else.

exit_ip, asn and geolocation are only filled in when check_url returns the exit IP, so pointing it at an ordinary web page leaves all three null. geolocation mirrors the GeoLite2 City record (abbreviated above); fields the database has no data for are omitted, and names are English only.

Stopping a run early, with ESC or q in the TUI or Ctrl-C anywhere, writes out everything checked so far. If you stop before any proxy has been checked, the previous run's files are left untouched rather than emptied.

Sponsors
IPcook.com

Want your name in this section? Support the project and it goes here.

Support this project

Star the repository so other people can find it. If you are interested in sponsoring, DM me on Telegram.

License

MIT

This product includes GeoLite2 Data created by MaxMind, available from https://www.maxmind.com

Nội dung README thuộc về monosans và cộng đồng đóng góp, giấy phép MIT. Nguồn: github.com/monosans/proxy-scraper-checker

Đăng nhập