Skip to content

chriskite/robotex

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Robotex

Obey Robots.txt

With one line of code, Robotex (pronounced like “robotics”) will download and parse the robots.txt file and let you know if your program is allowed to visit a given link.

Usage:

robotex = Robotex.new "My User Agent"
robotex.allowed?("http://www.example.com/foo")
robotex.delay!("http://www.example.com/foo") # wait until any specified Crawl-Delay has passed

Acknowledgements

Robotex is a modified version of Kyle Maxwell’s excellent Robots library. Some folks were unable to use that gem due to packaging issues, so I used his code to create Robotex.

About

Ruby library to obey robots.txt

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages