[Violet] + [Butterfly] = {StarStart!}
The World-Wide-Web crawler for Apache Solr
Get the execute file official build via GitHub Releases.
No installation required, but you can put it on the directory your terminal $PATH variable points for convenient.
Eventually,
set up your config file (Command: butterfly config),
and move it to the directory where you want to execute butterfly command.
butterfly -add <URI>
The <URI> is a variable, please fill in the URL you hope to fetch the website at first time.
It will be added into a list that the butterfly will fetch it.
butterfly list
It will display the URL list of websites that the butterfly will visit at the first time.
NOTICE: If the butterfly had been fetched the URL, it will ignore the URL next time.
But if you hope to re-fetch the website, please delete the database of this website, and restart the butterfly again.
butterfly -add <URI>
The <URI> is the variable that the website URL you want to remove from the list.
butterfly start
The butterfly will fly to fetch the websites in the list.
Butterfly requires GoLang >= 1.17
Please check your go version or install the latest version.
-
GNU Linux / MacOSX / Unix Like
Execute this command
sh bootstrap.sh -
Win32 Platform
We have no plan to support Microsoft Windows, but you still can get the go packages Butterfly required.
Just do it.
go build
The "butterfly" will be the execute file you build on your own self.
(c) 2020 Star Inc.