Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I have a website in which the code scrapes other websites for getting the accurate data. While the code works good but there a decent lag in performance because the code firsts downloads the html stream from various sites(some times 9 websites), extracts the relative part and then renders the html page.

What should I do to get an optimal performance. Should I change from shared hosting (godaddy) to my own server or it has nothing to do with my hosting and I need to make changes to my code?

share|improve this question
add comment

1 Answer

up vote 5 down vote accepted

API/CSV

Ask those websites if they provide an API, or, if you don't need an up-to-date information or the information you need doesn't change frequently, if they can sell/give you for free the data itself (for example in an CSV file). Some small websites may have fancier ways to access data, like a CSV file for the older information, and an RSS feed for the changed one.

Those websites would probably be happy to help you, since providing you with an API would reduce their own CPU and bandwidth usage by you.

Profile

Screen scrapping is really ugly when it comes to performance and scaling. You may be limited by:

  • your machine performance, since parsing, sometimes an invalid HTML file, takes time,
  • your network speed,
  • their network speed usage, i.e. how fast can you access the pages of their website depending on the restrictions they set, like the DOS protection and the number of requests per second for screen scrappers and search engine crawlers,
  • their machine performance: if they spend 500 ms. to generate every page, you can't do anything to reduce this delay.

If, despite your requests to them, those websites cannot provide any convenient way to access their data, but they give you a written consent to screen scrape their website, then profile your code to determine the bottleneck. It may be the internet speed. It may be your database queries. It may be anything.

For example, you may discover that you spend too much time finding with regular expressions the relevant information in the received HTML. In that case, you would want to stop doing it wrong and use a parser instead of regular expressions, then see how this improve the performance.

You may also find that the bottleneck is the time the remote server spends generating every page. In this case, there is nothing to do: you may have the fastest server, the fastest connection and the most optimized code, the performance will be the same.

Do things in parallel:

Remember to use parallel computing wisely and to always profile what you're doing, instead of doing premature optimization, in hope that you're smarter than the profiler.

Especially when it comes to using network, you may be very surprised. For example, you may believe that making more requests in parallel will be faster, but as Steve Gibson explains in episode 345 of Security Now, this is not always the case.

Legal aspects

Also note that screen scrapping is explicitly forbidden by the conditions of use (like on IMDB) on many websites. And if nothing is said on this subject in conditions of use, it doesn't mean that you can screen scrape those websites.

The fact that the information is available publicly on the internet doesn't give you the right to copy and reuse it this way neither.

Why? you may ask. For two reasons:

  • Most websites are relying on advertisement and marketing. When people use one of those websites directly, they waste some CPU/network bandwidth of the website, but in response, they may click on an ad or buy something sold on the website. When you screen scrape, your bot waste their CPU/network bandwidth, but will never click on an ad or buy something.

    Displaying the information you screen scrapped on your website can have even worse effects. Example: in France, there are two major websites selling hardware. The first one is easy and fast to use, has a nice visual design, better SEO, and in general is very well done. The second one is a crap, but the prices are lower. If you screen scrape them and give the raw results (prices with links) to your users, they will obviously click on the lower price every time, which means that the website with pretty design will have less chances to sell the products.

  • People made an effort in collecting, processing and displaying some data. Sometimes they paid to get it. Why would they enjoy seeing you pulling this data conveniently and for free?

share|improve this answer
    
I have no written consent from them but if the information is in public domain, then how can it be illegal to use the same. My website is a price-comparison engine. All I do is to take the price and linkup the page. If anyone can see the pricing information on their site, then how can it be illegal to use at my own end. It's just like a store argument. I am entitle to tell anyone what the price for xyz product is there. Anyways, do you think if I have my own server, then things might improve because my code is not great but its fine and there are no data queries involved. –  Pankaj Upadhyay Mar 25 '12 at 13:14
    
Lets say that you pull data from their site every second. They might not like the 86400 pulls you are doing each day. They may block your IP in order to keep their response times up. –  mhoran_psprep Mar 25 '12 at 21:14
    
@mhoran_psprep, that's a valid point. Sorry i didn't thought about that. But, I am not able to understand what can be done because the pricing changes so frequently and they don't have any API or XML feed therefore I had to come with such a code :-( –  Pankaj Upadhyay Mar 26 '12 at 5:40
    
@MainMa awesome answer mate. Thanks for your time and effort. Will there be any legal issues if i don't screen scrape their website, but manually enter the pricings in my database ?? –  Pankaj Upadhyay Mar 26 '12 at 13:46
    
@PankajUpadhyay: I would say yes, but I'm not a lawyer. The legislation may also be very different in India if the websites you are using are the websites hosted in your country. Consult a lawyer to have an answer you can rely on. –  MainMa Mar 26 '12 at 14:03
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.