Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I am researching ways to classify words in text and I'm wondering what options there are and which are best suited to this job. I'm mostly interested in keywords which are most often nouns.

So far I know I could use a Bayes classifier, blacklist, or whitelist. However, I haven't had the best of luck with each of these so far.

I started with a whitelist using the words provided by wordnet and moby to attempt to spot each noun. However, many words were missed as not existing in the database, or not being nouns.

Next I tried building a blacklist to match everything except known stop-words, verbs, and such. However, It would take a long investment of time to build a list large enough to handle the 200,000 known English (let alone other languages) words. In addition, keeping that many words in memory for comparison is not practical for performance on commodity hardware.

Using I have had luck with a bayes classifier matching spam and sentiment, but I'm not sure how to use it to tell nouns (or keywords) from other words since there are so many similarities between all the groups of words.

Last, I tried using regex to spot keywords based on proper nouns and names since they are always capitalized in English. The problem is that many keywords are not capitalized resulting in lacking results.

What other options exist for classifying keywords? What other design patterns could I use with the options above for better results?

share|improve this question
2  
Parts of Speech (POS) Tagging is an area I've dipped my toes in, but can't give any significant advice; I will point you however to the Association of Computer Linguistics matrix of POS state of the art test results aclweb.org/aclwiki/… and let you take it from there. –  JustinC Feb 13 '12 at 18:32
1  
Don't forget many words can be spelled the same way, and be a noun, or a verb depending on the context. –  CaffGeek Feb 13 '12 at 20:39
    
@Chad, yes. Take "want" for example. –  Xeoncross Feb 13 '12 at 20:47
    
A useful term that often comes up in CS and NLP literature is "OOV" (Out Of Vocabulary). –  hippietrail May 29 '13 at 7:00
add comment

2 Answers

up vote 1 down vote accepted

This is computational linguistics, and the book accompanying the open- source Natural Language Toolkit, Natural Language Processing with Python --- Analyzing Text with the Natural Language Toolkit, is available as a paper book and free online.

Chapter five covers "part-of-speech tagging" or POS-tagging with the NLTK in detail, using Python dictionaries:

As we have seen, a tagged word of the form (word, tag) is an association between a word and a part-of-speech tag. Once we start doing part-of-speech tagging, we will be creating programs that assign a tag to a word, the tag which is most likely in a given context. We can think of this process as mapping from words to tags. The most natural way to store mappings in Python uses the so-called dictionary data type (also known as an associative array or hash array in other programming languages). In this section we look at dictionaries and see how they can represent a variety of language information, including parts of speech.

It also covers several methods of automatically tagging text to identify POS in text with unknown words.

Some other introductory books to computational linguistics have ben suggested at Linguistics.SE, where you can ask further questions on the topic.

share|improve this answer
add comment

Possibly you are looking for Natural Language Processing?

To classify keywords, I would build up a corpus of sample text (or use an existing corpus). Then I would 'train' a natural language algorithm on your corpus. Then I would use that trained algorithm on whatever text you want to classify.

To get you started, here's a link to Apache's NLP project. And here is a book I've on Natural Language Processing with Python I've been eying.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.