Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm trying to automate the application of metadata to huge amount of text, but I'm not sure what language would make this task easier (if there is one).

What programming language is most suitable for handling unstructured text? Why? If the answer is "it depends", what are a few examples of why you would use one language over another?

share|improve this question

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

5  
Questions that start with "In your opinion, what's the best programming language", shall be punished by perl being the only answer... –  Yannis Rizos Nov 2 '11 at 0:45
4  
"Data mining, text analytics, natural language processing"... are all completely different applications. Could you please be more specific about what's in scope? –  Aaronaught Nov 2 '11 at 1:23
    
Automating the application of metadata to enormous amounts of text is the scope. –  measureallthethings Nov 2 '11 at 1:34
    
Whoa, what's this note on the bottom? –  Mehrdad Nov 2 '11 at 5:01

3 Answers 3

For natural language processing, a traditional fit is Prolog.

The book Prolog Programming in Depth has a single chapter devoted to using Prolog for NLP. According to this book, Prolog is a nice fit for NLP because of it's built-in knowledge representation system and unification mechanism.

According to Wikipedia (http://en.wikipedia.org/wiki/Prolog), :

While initially aimed at natural language processing, the language has since then stretched far into other areas

Prolog was initially developed to solve such problems.

As for data-mining the answer can depend on the format of the data. Web-pages, databases, text ? If you are data-mining information on text files, I would stay away from Prolog and use a general language you are comfortable with. C#, Python, etc. could do the job.


Data mining, text analytics, natural language processing...there're a lot of names for it.

The definition of NLP, which is according to Wikipedia:

Natural language processing (NLP) is a field of computer science and linguistics concerned with the interactions between computers and human (natural) languages; it began as a branch of artificial intelligence.

or by Prolog Programming in Depth:

Natural language processing (NLP) is the study of how to make computers understand human languages such as English and French.

While that has some common grounds with data mining, they are still two things very much apart in goal and in practice.

share|improve this answer
    
+1, clear and to the point. –  Emmad Kareem Nov 2 '11 at 4:20

It's hard to say what is best objectively.

My personal favourite for these kind of applications is Clojure, and here are the reasons why I think it is particularly strong on this space:

  • It's a Lisp - so you get all the power of "Beating the Averages". Basically Lisp combines the fact that is is homoiconic with a very sophisticated macro system that enables you to extend the language to your problem domain (via DSL and code generation). This is a huge productivity win once you have mastered it. Lisp also has a strong heritage in AI and symbolic computation (see this question for more)
  • It's a functional language first and foremost - FP is very well suited for applications that are really just a composition of many different data transformations. You tend to get very clean, concise and maintainable code for these sort of applications. The fact that Clojure is also lazy is a big bonus - it is conceptually very useful to be able to represent some streams of data as infinite lazy sequences for example, both in terms of clarity of code and the ability to handle larger-than-memory data sets.
  • Very good library availability - Clojure has a lot of people interested in AI / data processing / NLP etc, perhaps because of the Lisp heritage in these areas. For example, Incanter is a great R-like statistical computing library that makes it very easy to compute and visualise statistics on your data, and Pallet which is a simple tool for building clusters of compute machines (leveraging jclouds under the hood)
  • Access to Java ecosystem - this is really important - Java has the largest and most mature ecosystem of open source libraries for virtually any purpose on account of it's age, popularity, portability and suitability as a library language. You get access to all this for free from Clojure. As a result you can have your pick of great tools like Weka (for data mining), Cassandra (for your big data NoSQL needs). In my view, this advantage makes Clojure more valuable as a pragmatic language for "getting things done" than all other Lisps combined - library availability is never going to be an issue in Clojure.
  • Interactive development - the standard way to develop in Clojure is to interact with a live, running application environment via a REPL. This is surprisingly useful for data processing type applications - it's easy to test one-liners on your current data sets without having to restart or rebuild the application. This can save a vast amount of time when you are doing long-runnning computations....

Some mini code examples to give a flavour of the langauge and the kind of conciseness you can expect:

;; Frequency analysis (implicitly treating a string as a sequence of chars)   
(frequencies "abracadabra")
=> {\a 5, \b 2, \r 2, \c 1, \d 1}


;; Infinite, lazily calculated sequence of Fibonacci numbers
(def fibs
  (lazy-cat [0 1] (map + (rest fibs) fibs)))   
(take 10 fibs)
=> (0 1 1 2 3 5 8 13 21 34)


;; Plot a sin wave in Incanter:
(view (function-plot sin -10 10))
share|improve this answer
    
+1, Very comprehensive and organized answer. –  Emmad Kareem Nov 2 '11 at 4:20

Three candidates here:

  • Python would be my favourite. It has superb string handling and a vast number of libraries to help you parse text. It also has superb support for structures (Dictionaries,linked lists etc.) which ironically you will really need when trying to make sense of unstructured data.
  • perl for much the same reason as Python, the language itself can be a little more awkward but then it has even more ready built libraries and modules for parsing text.
  • Lua would be an off the wall choice. The languages built in text handling is great and the beguilingly simple syntax make it a joy to program, but, alas, third party libraries are somewhat lacking so you may need to re-invent a wheel or two.
share|improve this answer
    
Plus Python has NLTK. I haven't found a library that comprehensive yet. –  sebastiangeiger Nov 2 '11 at 7:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.