Tell me more ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
up vote 2 down vote favorite

I'm trying to build a very small, niche search engine, using Nutch to crawl specific sites. Some of the sites are news/blog sites. If I crawl, say, techcrunch.com, and store and index their frontpage, then within hours my index for that page will be out of date.

Does a large search engine such as Google have an algorithm to re-crawl frequently updated pages very frequently, hourly even? Or does it just score frequently updated pages very low so they don't get returned?

Also, how can I handle this in my own index?

share|improve this question
Interestingly enough, Matt Cutts (head of Google webspam team) talked about this kind of thing literally a couple of days ago. Only a brief overview, but it may be of some use to you - searchengineland.com/… – Anonymous Apr 26 '12 at 12:23
It's all statistics based. The engine continuously updates its database dedicating time proportional to its popularity, meaning unclicked sites are updated rarely while latest news sites are always up to date. If I were you, I would average the proposed expiration time with the click rate or something of that sort, but then it's up to you. – Neil Apr 26 '12 at 15:52
@Neil Understood. My engine is really simple and at the moment I don't have the time nor resources to crawl frequently enough to have the freshest content. For it's intended purpose, my engine also doesn't need the freshest content. However, it's important that it NOT point to pages with content that's changed since my indexing. I've come up with one easy (although very imperfect) solution: Count the number of /'s in the url and boost pages with higher / counts. It's a horrible solution, but for the major blogs and news sites I'm crawling, this might work with their deep site structure. – Smooth Almonds Apr 26 '12 at 21:45
Why haven't you given expiration time more thought? It's normally used to inform browsers when they can no longer use what is contained in cache but would have to again request from the server since it's likely changed since then. Of course, some sites have negative expiration times, meaning you must update always. You likely don't want to invalidate the page the moment you cache it, so you should find some sort of compromise, say a minimum expiration time of a day if proposed expiration time is less. – Neil Apr 27 '12 at 8:36

1 Answer

up vote 1 down vote

Yes, Google does manage the frequency of how much time it's crawlers are visiting a site. For example all the news site which are registered on Google News get more frequent visits than other sites, also some live-score showing websites get crawler's vistit every second(realtime search). So, it means they are using some kind of algorithm so that those bots prioritise their visits to such sites.

share|improve this answer
See also what google has to say (not much): support.google.com/webmasters/bin/… – George Duckett Apr 26 '12 at 11:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.