crawlr 0.2.1

Crawlr is a modern Ruby web scraping framework built for respectful and efficient data extraction. Features async HTTP requests, robots.txt compliance, cookie management, proxy rotation, flexible CSS/XPath selectors, and comprehensive error handling. Designed for both simple scraping tasks and complex, large-scale data extraction projects.

Gemfile:
=

install:
=

Versions:

0.2.3 October 03, 2025 (31.5 KB)
0.2.2 October 01, 2025 (31.5 KB)
0.2.1 September 30, 2025 (31 KB)
0.2.0 September 29, 2025 (31 KB)
0.1.0 September 29, 2025 (30.5 KB)

Runtime Dependencies (5):

async ~> 2.32

async-http ~> 0.91.0

concurrent-ruby ~> 1.3

http-cookie ~> 1.0

nokogiri ~> 1.18

Development Dependencies (5):

rake ~> 13.0

rspec ~> 3.0

rubocop ~> 1.60

ruby-prof ~> 1.7

webmock ~> 3.25

Show all transitive dependencies

Owners:

Pushed by:

Authors:

Aristotelis Rapai

SHA 256 checksum:

Total downloads 1,006

For this version 218

Version Released:
September 30, 2025 10:05pm

License:
MIT

Required Ruby Version: >= 3.1.0

New versions require MFA: true

Version published with MFA: true

Links:

Homepage Changelog Bug Tracker Download Review changes Badge RSS Report abuse Reverse dependencies