Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm currently working on a webscraper that is going to be processing about 2 million pages in total. But I'm not entirely sure how to benchmark it.

Currently it processes about 100,000 pages in an hour at about 2mb/s. It isn't saturating the the bandwidth of the server but I've tried a lot of ways to make it quicker but to no avail. CPU is running 50% or so on average.

share|improve this question
How often do you need to scrape the pages? – John Dec 18 '10 at 4:34
If the target web site you are scraping has done their homework, you won't be able to scrape more than a page or two every second anyway. – Robert Harvey Dec 18 '10 at 4:40
If someone were to poll my server for content at that rate I'd block him as a potential DOS attack... – jwenting Mar 29 '11 at 13:24
You are saying it's not saturating the bandwidth of the server like it's a bad thing... – Gilles Apr 5 '11 at 12:58
up vote 2 down vote accepted

That means you can get a fresh copy of every page roughly every day? Is that fast enough for you? If it's not, profile and improve.

share|improve this answer

Whose pages are you scraping, and for what reason?

You're talking about roughly thirty page requests per second, or 2 MB/s. You say it's not saturating the bandwidth of "the" server - is that your server or theirs? Saturating the bandwidth of somebody else's server is generally considered to be a denial of service attack.

Two megabytes a second is a lot of somebody else's bandwidth to be using. I couldn't do that from home, as my broadband connection is simply not that fast. Some web hosting services have bandwidth caps, meaning you'd be shutting people's sites down for a certain period of time.

If the pages you're scraping belong to somebody big enough not to mind this, check to see if they've got an API you can use instead. If they don't, consider cutting back so you're not perceived as an attacker.

share|improve this answer
I'm wondering myself if this really a web scraper... – Gilles Apr 5 '11 at 12:55

What is a good speed for a webscraper?

That's impossible to answer. It depends on many things; e.g.

  • How rate at which the site you are scraping can (or is willing to) handle your requests.
  • The effective network latency and bandwidth when fetching web pages.
  • How much work your machine has to do to process each scraped page and record the information you extracted.

If you are running at 50% CPU, you may be able to do better, or you may not. For instance, if your requests are being throttled / limited by the remote server, or if that's as fast as it can go, then you are stuck. If you are running into network latency / bandwidth issues then you may need to upgrade your network connection ... or move to a data centre that is nearer the site you are trying to scrape.

share|improve this answer

Seems like you need to go to the next level and do "Distributed" scraping.

You are going to need to build your application to be able to scale out to multiple servers.

The challenge will be somehow managing the the separate notes in your scraping "Grid".

You could look at centralised head server that dispatches work out to the nodes.

Other things to avoid would be two or more servers scraping the same site (this is duplication that is simply not needed).

share|improve this answer
+1: After the software's been profiled for bottlenecks and there really is no obvious way to make it go faster, this will be an excellent solution. – doppelgreener Dec 19 '10 at 8:02
you could use consistent hashing to work out which node should crawl which pages. – dan_waterworth Dec 19 '10 at 8:51
Be careful of the fine line between distributed scraping and distributed denial of service. – David Thornley Mar 29 '11 at 13:58
@David Thornley: Which line ? When you are talking about distributed scraping and distributed is between "", it's pretty clear it's not scraping anymore. Even if it is, the server getting scraped will be on it's knees for all pratical purposes. – Gilles Apr 5 '11 at 12:56

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.