Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

What sort of languages would be able to handle writing your own web crawler?

Could PHP handle this? I'm quite good with PHP (following best practices etc).

But I'd like a good reason to learn a new language if I need to.

The idea is to crawl one specific website that has multiple entries, much like an RSS feed, but they don't offer that an RSS feed of the site...

share|improve this question
6  
Don't forget to download the robots.txt file and obey the site restrictions and crawling speed limits. Websites can get quite annoyed when you go faster and start banning you. en.wikipedia.org/wiki/… –  Loki Astari Aug 2 '11 at 2:04

2 Answers 2

up vote 4 down vote accepted

Any language that has good built-in HTTP facilities for downloading and parsing (badly formed) web documents would be fine. I'd recommend a scripting language like Python, because a webcrawler generally spends most of its time waiting on network I/O (DNS or HTTP responses), so performance is generally not much of a concern. Python also has decent libraries for managing HTTP connections and parsing HTML. You need a library specifically designed to handle badly formed HTML, because the web is mostly crappy HTML. If you choose Python, you can use something like libxml2dom.

share|improve this answer
4  
Wait, what, performance is not a concern? Sure, you'll be doing a lot of I/O waits, but you'll also probably be processing 50 requests in parallel. –  Aaronaught Aug 2 '11 at 2:13
    
Even with 100 parallel requests using a combination of async I/O and multiple threads, a typical webcrawler spends the majority of its time waiting for a response from remote webservers. –  Charles Salvia Aug 2 '11 at 5:16
1  
Well, you're correct of course that it depends on what you're doing with each fetched document. But unless you're doing something incredibly CPU-intensive with each download, you're not likely to be CPU bound for any significant percentage of the crawling time. In practice the CPU will be sitting idle a lot, even with hundreds of parallel requests, because the crawler not only has to wait for responses from each remote server, but also implement artificial wait times for politeness purposes, e.g. wait 1-5 seconds per request per host. –  Charles Salvia Aug 2 '11 at 13:39
1  
So, unless you're pounding a single host over and over, you're not likely to be CPU bound for any significant portion of the webcrawl. Even a commercial crawler that fetches from thousands of hosts simultaneously will wait a long time, since most hosts have few documents and the crawler will eventually end up slowly fetching docs from a few large hosts. That's been my experience, at least. –  Charles Salvia Aug 2 '11 at 13:40
1  
@Aaronaught: I write a crawler for my company. IO is definitely the primary bounding factor (even without DNS lookup done by a separate scheduling processes). Parsing and correcting for HTML is trivial in comparison. We spin up-to max 20,000 parallel requests in each crawler and have 10 crawlers running simultaneously. And yes it can go very paralleled, but you must obey the speed-limits on each site (and even if they do not have explicit speed-limits we make sure we do not hit sites hard (as we don't want to banned)). Note: multiplex over 20,000 sites is hard so this limit is rarely reached. –  Loki Astari Aug 2 '11 at 14:33

I would recommend you using python, but that's because I only know how to -code- a crawler in python and Java. And I would definitely not recommend you going 5 feet near Java for that task.

Take a look at:
Mechanize - good tool for web browsing
Scrapy - good framework to screen scraping and web crawling

A good reason is that you already have some frameworks built in python just for that.

share|improve this answer
    
I agree, Python is going to be the fastest route to something working that doesn't explode on strange input. –  Tim Post Aug 2 '11 at 7:28
    
Recommendation based on the fact that is the only language you know is not a good recommendation. Recommendations require that you have done it in many languages and know the pros/cons for each. –  Loki Astari Aug 2 '11 at 14:40
    
that's why I added the comment... And I've done it in ruby, java, python and c#. But I've REALLY done it in Java and Python. If you go around saying that you're the master of the universe and know how to code it properly in all n languages.. Yeah.. People is going to believe in it... –  wleao Aug 2 '11 at 14:46
    
Scrapy looks nice. Just saying –  MattyD Aug 2 '11 at 22:58

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.