Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm talking the kind of search that auto-suggests your query as you type, the way Google does, the way Wikipedia does, the way Stack Exchange suggests other questions as you type the title, etc. And Wikipedia used to have an awful search implementation, for those of us who remember. It had no auto-suggest, was shut down by the slightest typo or misspelling, and rarely gave pertinent results.

As an amateur web programmer, I'm trying to figure out how I would implement an effective and high-performing search-and-auto-suggest as Wikipedia now has, and as SE has, and as so many other websites have.

How can you obtain that kind of incredibly fast performance and flexibility? Does everyone just wing it, or are there principles, guidelines, helpful libraries, or other such resources to help you get it done right in the style of the "Internet of the Future"?

share|improve this question
Should this be a StackOverflow question? Sometimes I can't tell. – Aerovistae Jan 24 '13 at 8:22
no, I don't think so. It's not specific enough for SO (roughly: can't be answered with code) – thorsten müller Jan 24 '13 at 8:25
Chances are the answers to this question do equally well apply if you produce a search for an application that stores a lot of data. It's probably not restricted to websites. – Joachim Sauer Jan 24 '13 at 9:41
Seems like a question for – JohnL Jan 24 '13 at 10:22
Ironically, I've never been crazy about the look and feel of that site... Feels 'over-designed' or something. – Graham Jan 24 '13 at 14:21
up vote 6 down vote accepted

For a start you will need a full text search engine like Apache Solr or Sphinx (there are more and some databases have full text features too, but I know those two and they are free and work great). If it has facet search (like Solr) this will help a lot (for certain types of queries). This will cover the largest part of indexing and performance issues.

From there on you need to analyze your data and the search needs of your customers and come up with a good setup of fields, field types, field weights, and the text search specific issues like stemming or multiple languages, handling of special characters, splitting compound words into single words (very important for german search, we can chain words arbitrarily here).

In addition you can do a lot of statistics like counting words or phrases your users search for and compare them against statistics of your index data (for example to find out what synonyms are relevant for you)

share|improve this answer
Stemming is probably the most important thing that full text search engines do that your typical database usually doesn't do: "check", "checking", checked" and "checker" will be identified as related (or even identical) and searching for one will find the other. That makes a huge difference in the usability of a search. – Joachim Sauer Jan 24 '13 at 9:40
Some database engines do stemming - the full text indexing in MS SQL Server does that (… - look for the FORMSOF keyword, I think). WHether it does the best job of it, I don't know. – JohnL Jan 24 '13 at 10:26

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.