Available options for classifying words in text?

Question

I am researching ways to classify words in text and I'm wondering what options there are and which are best suited to this job. I'm mostly interested in keywords which are most often nouns.

So far I know I could use a Bayes classifier, blacklist, or whitelist. However, I haven't had the best of luck with each of these so far.

I started with a whitelist using the words provided by wordnet and moby to attempt to spot each noun. However, many words were missed as not existing in the database, or not being nouns.

Next I tried building a blacklist to match everything except known stop-words, verbs, and such. However, It would take a long investment of time to build a list large enough to handle the 200,000 known English (let alone other languages) words. In addition, keeping that many words in memory for comparison is not practical for performance on commodity hardware.

Using I have had luck with a bayes classifier matching spam and sentiment, but I'm not sure how to use it to tell nouns (or keywords) from other words since there are so many similarities between all the groups of words.

Last, I tried using regex to spot keywords based on proper nouns and names since they are always capitalized in English. The problem is that many keywords are not capitalized resulting in lacking results.

What other options exist for classifying keywords? What other design patterns could I use with the options above for better results?

Answer 1

This is computational linguistics, and the book accompanying the open- source Natural Language Toolkit, Natural Language Processing with Python --- Analyzing Text with the Natural Language Toolkit, is available as a paper book and free online.

Chapter five covers "part-of-speech tagging" or POS-tagging with the NLTK in detail, using Python dictionaries:

As we have seen, a tagged word of the form (word, tag) is an association between a word and a part-of-speech tag. Once we start doing part-of-speech tagging, we will be creating programs that assign a tag to a word, the tag which is most likely in a given context. We can think of this process as mapping from words to tags. The most natural way to store mappings in Python uses the so-called dictionary data type (also known as an associative array or hash array in other programming languages). In this section we look at dictionaries and see how they can represent a variety of language information, including parts of speech.

It also covers several methods of automatically tagging text to identify POS in text with unknown words.

Some other introductory books to computational linguistics have ben suggested at Linguistics.SE, where you can ask further questions on the topic.

Answer 2

Possibly you are looking for Natural Language Processing?

To classify keywords, I would build up a corpus of sample text (or use an existing corpus). Then I would 'train' a natural language algorithm on your corpus. Then I would use that trained algorithm on whatever text you want to classify.

To get you started, here's a link to Apache's NLP project. And here is a book I've on Natural Language Processing with Python I've been eying.