Follow robots.txt yes/no

[eklem]

-f --followrobotstxt <yes/no> if you want your fetcher to play nice or not

[eklem]

I guess there are two things to check for.
1: User agent and if it matches specific or * is used.
2: Make an array of parts of site to not follow and check each link that the crawler wants to follow against this array

[eklem]

And default to "yes". The user-agent string connects to this, but it's not necessary to develope this one.
#10