Sign up ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free.

I'm trying to automate the application of metadata to huge amount of text, but I'm not sure what language would make this task easier (if there is one).

What programming language is most suitable for handling unstructured text? Why? If the answer is "it depends", what are a few examples of why you would use one language over another?

share|improve this question

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

closed as off-topic by gnat, Snowman, GlenH7, durron597, Kilian Foth Oct 27 at 6:59

This question appears to be off-topic. The users who voted to close gave these specific reasons:

  • "Questions asking us to recommend a tool, library or favorite off-site resource are off-topic for Programmers as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it." – Snowman, Kilian Foth
  • "Questions about what language, technology, or project one should take up next are off topic on Programmers, as they can only attract subjective opinions for answers. There are too many individual factors behind the question to create answers that will have lasting value. You may be able to get help in The Whiteboard, our chat room." – gnat, GlenH7, durron597
If this question can be reworded to fit the rules in the help center, please edit the question.

Questions that start with "In your opinion, what's the best programming language", shall be punished by perl being the only answer... – Yannis Nov 2 '11 at 0:45
"Data mining, text analytics, natural language processing"... are all completely different applications. Could you please be more specific about what's in scope? – Aaronaught Nov 2 '11 at 1:23
Automating the application of metadata to enormous amounts of text is the scope. – measureallthethings Nov 2 '11 at 1:34
Whoa, what's this note on the bottom? – Mehrdad Nov 2 '11 at 5:01

3 Answers 3

It's hard to say what is best objectively.

My personal favourite for these kind of applications is Clojure, and here are the reasons why I think it is particularly strong on this space:

  • It's a Lisp - so you get all the power of "Beating the Averages". Basically Lisp combines the fact that is is homoiconic with a very sophisticated macro system that enables you to extend the language to your problem domain (via DSL and code generation). This is a huge productivity win once you have mastered it. Lisp also has a strong heritage in AI and symbolic computation (see this question for more)
  • It's a functional language first and foremost - FP is very well suited for applications that are really just a composition of many different data transformations. You tend to get very clean, concise and maintainable code for these sort of applications. The fact that Clojure is also lazy is a big bonus - it is conceptually very useful to be able to represent some streams of data as infinite lazy sequences for example, both in terms of clarity of code and the ability to handle larger-than-memory data sets.
  • Very good library availability - Clojure has a lot of people interested in AI / data processing / NLP etc, perhaps because of the Lisp heritage in these areas. For example, Incanter is a great R-like statistical computing library that makes it very easy to compute and visualise statistics on your data, and Pallet which is a simple tool for building clusters of compute machines (leveraging jclouds under the hood)
  • Access to Java ecosystem - this is really important - Java has the largest and most mature ecosystem of open source libraries for virtually any purpose on account of it's age, popularity, portability and suitability as a library language. You get access to all this for free from Clojure. As a result you can have your pick of great tools like Weka (for data mining), Cassandra (for your big data NoSQL needs). In my view, this advantage makes Clojure more valuable as a pragmatic language for "getting things done" than all other Lisps combined - library availability is never going to be an issue in Clojure.
  • Interactive development - the standard way to develop in Clojure is to interact with a live, running application environment via a REPL. This is surprisingly useful for data processing type applications - it's easy to test one-liners on your current data sets without having to restart or rebuild the application. This can save a vast amount of time when you are doing long-runnning computations....

Some mini code examples to give a flavour of the langauge and the kind of conciseness you can expect:

;; Frequency analysis (implicitly treating a string as a sequence of chars)   
(frequencies "abracadabra")
=> {\a 5, \b 2, \r 2, \c 1, \d 1}

;; Infinite, lazily calculated sequence of Fibonacci numbers
(def fibs
  (lazy-cat [0 1] (map + (rest fibs) fibs)))   
(take 10 fibs)
=> (0 1 1 2 3 5 8 13 21 34)

;; Plot a sin wave in Incanter:
(view (function-plot sin -10 10))
share|improve this answer
+1, Very comprehensive and organized answer. – NoChance Nov 2 '11 at 4:20
I disagree. But I'm going you a +1 for a strong argument. – Tom Dworzanski Oct 9 at 2:08

For natural language processing, a traditional fit is Prolog.

The book Prolog Programming in Depth has a single chapter devoted to using Prolog for NLP. According to this book, Prolog is a nice fit for NLP because of it's built-in knowledge representation system and unification mechanism.

According to Wikipedia (, :

While initially aimed at natural language processing, the language has since then stretched far into other areas

Prolog was initially developed to solve such problems.

As for data-mining the answer can depend on the format of the data. Web-pages, databases, text ? If you are data-mining information on text files, I would stay away from Prolog and use a general language you are comfortable with. C#, Python, etc. could do the job.

Data mining, text analytics, natural language processing...there're a lot of names for it.

The definition of NLP, which is according to Wikipedia:

Natural language processing (NLP) is a field of computer science and linguistics concerned with the interactions between computers and human (natural) languages; it began as a branch of artificial intelligence.

or by Prolog Programming in Depth:

Natural language processing (NLP) is the study of how to make computers understand human languages such as English and French.

While that has some common grounds with data mining, they are still two things very much apart in goal and in practice.

share|improve this answer
+1, clear and to the point. – NoChance Nov 2 '11 at 4:20

Three candidates here:

  • Python would be my favourite. It has superb string handling and a vast number of libraries to help you parse text. It also has superb support for structures (Dictionaries,linked lists etc.) which ironically you will really need when trying to make sense of unstructured data.
  • perl for much the same reason as Python, the language itself can be a little more awkward but then it has even more ready built libraries and modules for parsing text.
  • Lua would be an off the wall choice. The languages built in text handling is great and the beguilingly simple syntax make it a joy to program, but, alas, third party libraries are somewhat lacking so you may need to re-invent a wheel or two.
share|improve this answer
Plus Python has NLTK. I haven't found a library that comprehensive yet. – sebastiangeiger Nov 2 '11 at 7:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.