Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I am working on a project that requires that I understand different techniques used by search engines for the web.

I have a strong scientific and development background, so I am not afraid of highly technical information.

I am looking for all forms of technical information including information on web crawlers, other techniques for acquiring data, methods of data storage and how to query it, etc., etc.

I am completely new to the subject and I'm looking for useful resource and books on the subject. Any suggestions are appreciated.

share|improve this question
why not ask search engine itself ;-) , Type "How does google work" and you will know. –  Pankaj Upadhyay Sep 3 '11 at 18:09
Steven Levy's new book on Google, In The Plex, is an in-depth and fascinating look at the company. It also covers the four main duties of a search engine, and how Google innovated in all go those dimensions. –  Macneil Sep 3 '11 at 18:21
@Pankaj : that's true, there is a lot of resource online ... but it is quite sparse, and I am a bit lazy to gather everything to put the pieces together. If a good book existed, that would be great! And also, I guess there must be different techniques ... Google is of course "the guide" when it comes to search engines, but I want to learn about many techniques and options so I can choose the best ... –  sebpiq Sep 3 '11 at 19:14
@Macneil: it seems very interestant indeed ! I might read it ... however it doesn't seem to talk about it from the technologic angle at all ... –  sebpiq Sep 3 '11 at 19:14
add comment

closed as off-topic by gnat, GlenH7, MichaelT, World Engineer Sep 16 '13 at 21:40

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions asking us to recommend a tool, library or favorite off-site resource are off-topic for Programmers as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it." – gnat, GlenH7, MichaelT, World Engineer
If this question can be reworded to fit the rules in the help center, please edit the question.

3 Answers

up vote 2 down vote accepted

This area of study is known as Information Retrieval. This Wikipedia article contains a good summary and lots of useful links.

share|improve this answer
+1 having studied my masters in Information Retrieval, a "search engine" is just the tip of the iceberg –  Darknight Sep 3 '11 at 21:46
I read the wikipedia article, and that looks like good theoretical information ! What I was looking for ! –  sebpiq Sep 3 '11 at 22:24
Any good reference about "information retrieval" then ? @Darknight what books did you study from ? –  sebpiq Sep 3 '11 at 23:13
add comment

First, Google actively participates in the "science" of technology and often shares their knowledge by releasing papers from their R&D department. You can find those papers from the reference link below. I haven't searched for specific papers on search retrieval algorithms but there should be more than enough information available on the subject from a very technical perspective, as well as papers on storing massive sets of data and effectively querying it.

Publications by Googlers

Other than Google's resources, I highly recommend that you look into Semantic Web research. Semantic web isn't a method of searching data and even though the concept at first seems a little vague, the clear implications of a semantic web "engine" would be to parse the information within the WWW and link relevant information with one another.

In short, semantic web is the science of what many forward-thinkers hope and are working for the internet to truly become where the information that is provided is well parsed, interpreted and correctly linked together. I haven't looked into it much myself, so some of my information may be a bit incorrect. However, their are plenty of resources available that discuss semantic web and many are hoping/waiting/working on a break through within the field with the hopes of making the "next big thing" for the internet.

A good starting point for learning about semantic web is, of course, Wikipedia.

These references may not be books but it is a lot of information. Reading and sifting through all of the technical information should keep you busy for a while.

share|improve this answer
@Thanks a lot ! This link to google articles is great ! I didn't know that such papers existed, and free to consult !!! About the semantic web, I knew a bit about the principles already, I will take a look at it, but from what I remember it is something that has been a dream for 10 years, but is still far from being reality (I might be wrong), ... but thanks again !!! –  sebpiq Sep 3 '11 at 21:08
Yes it is a bit of a dream, but while people dream, a lot of research is done to try and make it happen. I'll admit, it is a bit over my head but to my understanding, it is the 'hope' of the future, so it's kind of the cutting edge of the field. –  RLH Sep 3 '11 at 21:17
Yes ... you are right ! I probably need to take a look at it, at least for the sake of culture ! –  sebpiq Sep 3 '11 at 21:30
-1 although you answer is not "incorrect", the OP asked about the "theories" behind google/search engines. This area of study is know as Information Retrieval and Natural Language Processing (NLP). as answered Andy Waite –  Darknight Sep 3 '11 at 21:49
add comment

Following advices from @Andy Waite I read the wikipedia page on information retrieval, and followed the references. There is a lot of online information, and I find this introduction to information retrieval, which is an online book from 2008, so probably up-to date, and which seems to be quite a good intro to the topic.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.