Tell me more ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
up vote 4 down vote favorite
1

I started building a web crawler and read somewhere that it's a very hard problem to find a good seed page for the crawler. Can anyone explain me if there is any pre-defined procedure/ guidlines of finding a good seed page? or how you say that a particular page is a good seed page?

share|improve this question
2  
I have no actual experience on this, but I'd probably start with wikipedia or something like that. – tdammers Feb 1 at 12:04
This question may be better suited to the webmaster's stack exchange site seo tag (and it may already be answered there too). – MichaelT Feb 1 at 14:25

closed as off topic by Blrfl, Glenn Nelson, Martijn Pieters, MichaelT, GlenH7 Feb 1 at 15:07

Questions on Programmers Stack Exchange are expected to relate to software development within the scope defined in the FAQ. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about closed questions here.

2 Answers

up vote 3 down vote accepted

A good seed page needs to have

  • As many links as possible
  • For as many different topics as possible

That's all, really. The first things that come to mind would be Wikipedia and the Open Directory Project.

share|improve this answer
up vote -1 down vote

Since I believe generalistic web search should be now considered a solved problem with a single provider that has such an headstart, and momentum, that it's unpractical, unconvenient, and kinda lame to even consider outrunning it, I believe you might prefer focusing your search efforts on a specialized domain.

e.g.: U.S. law, E.U. law, local laws of certain foreign countries, documents in a certain language, fishing, hunting, the military, ambientalism, indian car manifacturers, south american team games, russian motorsports...

Specializing your domains is all about finding the appropriate seeds for the given domain. (Also, IMHO, a handul of selected, unknowing, specialized forums should also be considered as a starting point) So once you have you domain, if you dig it well, (or at least the one you're working with/for does) the seeds should come natural.

Otherwise head for dmoz.org like Google did some 15 years ago. (And don't forget to weight in yahoo.com for good measure) Give your crawler another 15 years and you should have a good link base to search upon.

Ok, ok, if you really really must take this hard and pointeless route also consider: reddit.com, twitter.com, and (why not) the stackexchange.com constellation.

share|improve this answer
he's not trying to compete with google, it's probably just a for-fun project – Clash Feb 1 at 14:39
Sites where users post masses of content including many links would be a good choice - I think they should links to a diverse range of sites. So I definitely agree with your reddit, twitter, stackexchange suggestion. – Carson63000 Feb 1 at 20:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.