WARNING: This software is new, experimental, and under heavy development. The documentation is lacking, if any. There are almost no tests.
You have been warned
Package crawler provides a load-balanced, concurrent and flexible crawler that follows the robots.txt policies and crawl delays in its default configuration.
To install, simply run in a terminal:
go get github.com/mars9/crawler