parolkar-obey_robots_dot_txt 1.0.0

Easy to use extension of Net::HTTP to let you obey robots.txt while crawling/scrapping/mining

Gemfile:
=

install:
=

Versions:

1.0.0 August 10, 2014 (9 KB)

Development Dependencies (1):

hoe >= 2.3.2

Show all transitive dependencies

Authors:

Abhishek Parolkar

SHA 256 checksum:

=