Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

When you go to the Appendix D in Oracle Text Reference they provide lists of stopwords used by Oracle Text when indexing table contents.

When I see the English list, nothing puzzles me. But the reason why the French list includes moyennant (French for in view of which) for example is unclear. Oracle has probably thought it through more than once before including it.

How would you constitute a list of appropriate stopwords if you were to design an indexer?

share|improve this question
add comment

2 Answers

Haven't really tried to build up a stopword list before, but my approach would be statistical analysis of appearance of words in text corpus.

Common words, which appear in many documents all over corpus with similar probability (and maybe low standard deviation), are bad at "identifying" any given document, should go into list of stopwords.

Vice versa, rare words are excellent at "identifying" documents (and are good keywords and search terms), and should not go in stopword list.

share|improve this answer
add comment

People don't usually search on articles, pronouns, prepositions, conjunctions and the like...adding them to your index just adds noise. I would also include common phrases that don't really add meaning to the text (whereas, instead of, on the other hand). Moyennant appears to fall in the latter category, that's probably why it's included as a stop word.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.