Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to handle and process ~5 million news articles, an amount small enough to fit on a commodity desktop but too large to process serially. Thus far, I have been using Python/pickle for serialization, but this is easily getting out of hand. However, I do not have a server much less a cluster immediately available for distributed storage.

What tools can I use to store, query, and operate on this size of a dataset? The majority of operations I need to do can be formulated as Map operations; the only exception is that the Natural Language Processing portion which is handled by calling Stanford's CoreNLP software suite (it needs a few seconds to load some large parameter files).

share|improve this question
Your doing this with Python? – Dynamic Apr 22 '12 at 23:38

Many people speak highly of Solr. It specializes in searching large quantities of text, and should be usable from Python.

I suppose you could also do something custom with Hadoop. It directly supports map reduce operations, though it might be easier to work in Java than Python with Hadoop.

share|improve this answer
Solr is good... :) – c0da May 7 '12 at 5:18

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.