I started building a web crawler and read somewhere that it's a very hard problem to find a good seed page for the crawler. Can anyone explain me if there is any pre-defined procedure/ guidlines of finding a good seed page? or how you say that a particular page is a good seed page?
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
closed as off topic by Blrfl, Glenn Nelson, Martijn Pieters, MichaelT, GlenH7 Feb 1 '13 at 15:07
Questions on Programmers Stack Exchange are expected to relate to software development within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.
A good seed page needs to have
That's all, really. The first things that come to mind would be Wikipedia and the Open Directory Project.
Since I believe generalistic web search should be now considered a solved problem with a single provider that has such an headstart, and momentum, that it's unpractical, unconvenient, and kinda lame to even consider outrunning it, I believe you might prefer focusing your search efforts on a specialized domain.
Specializing your domains is all about finding the appropriate seeds for the given domain. (Also, IMHO, a handul of selected, unknowing, specialized forums should also be considered as a starting point) So once you have you domain, if you dig it well, (or at least the one you're working with/for does) the seeds should come natural.
Otherwise head for dmoz.org like Google did some 15 years ago. (And don't forget to weight in yahoo.com for good measure) Give your crawler another 15 years and you should have a good link base to search upon.