Besides obeying robots.txt, obey
<meta> elements and links:
There are many who believe robots.txt is not the proper way to block indexing and because of that viewpoint, have instructed many site owners to rely on the
<meta name="robots" content="noindex"> tag to tell web crawlers not to index a page.
If you're trying to make a graph of connections between websites (anything similar to PageRank),
<meta name="robots" content="nofollow">) is supposed to indicate the source site doesn't trust the destination site enough to give it a proper endorsement. So while you can index the destination site, you ought not store the relation between the two sites.
SEO is more of an art than a real science, and it's practiced by a lot of people who know what they're doing, and a lot of people who read the executive summaries of people who know what they're doing. You're going to run into issues where you'll get blocked from sites for doing things that other sites found perfectly acceptable due to some rule someone overheard or read in a blog post on SEOmoz that may or may not be interpreted correctly.
Because of that human element, unless you are Google, Microsoft, or Yahoo!, you are presumed malicious unless proven otherwise. You need to take extra care to act as though you are no threat to a web site owner, and act in accordance with how you would want a potentially malicious (but hopefully benign) crawler to act:
- stop crawling a site once you detect you're being blocked: 403/401s on pages you know work, throttling, time-outs, etc.
- avoid exhaustive crawls in relatively short periods of time: crawl a portion of the site, and come back later on (a few days later) to crawl another portion. Don't make parallel requests.
- avoid crawling potentially sensitive areas: URLs with
/admin/ in them, for example.
Even then, it's going to be an up-hill battle unless you resort to black-hat techniques like UA spoofing or purposely masking your crawling patterns: many site owners, for the same reasons above, will block an unknown crawler on sight instead of taking the chance that there's someone not trying to "hack their site". Prepare for a lot of failure.
One thing you could do to combat the negative image an unknown crawler is going to have is to make it clear in your user-agent string who you are:
Aarobot Crawler 0.9 created by John Doe. See http://example.com/aarobot.html for more information.
http://example.com/aarobot.html explains what you're trying to accomplish and why you're not a threat. That page should have a few things:
- Information on how to contact you directly
- Information about what the crawler collects and why it's collecting it
- Information on how to opt-out and have any data collected deleted
That last one is key: a good opt-out is like a Money Back Guarantee™ and scores an unreasonable amount of goodwill. It should be humane: one simple step (either an email address or, ideally, a form) and comprehensive (there shouldn't be any "gotchas": opt-out means you stop crawling without exception).