Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.
Required Ruby Version
>= 0
Authors
Stewart McKee
Versions
- 1.2.1 January 09, 2021 (9.28 MB)
- 1.2.0 February 25, 2019 (9.28 MB)
- 1.1.0 November 06, 2015 (9.28 MB)
- 1.0.29 November 05, 2015 (9.28 MB)
- 1.0.28 March 20, 2015 (9.28 MB)
- 1.0.25 January 24, 2015 (9.28 MB)