Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I need to handle and process ~5 million news articles, an amount small enough to fit on a commodity desktop but too large to process serially. Thus far, I have been using Python/pickle for serialization, but this is easily getting out of hand. However, I do not have a server much less a cluster immediately available for distributed storage.

What tools can I use to store, query, and operate on this size of a dataset? The majority of operations I need to do can be formulated as Map operations; the only exception is that the Natural Language Processing portion which is handled by calling Stanford's CoreNLP software suite (it needs a few seconds to load some large parameter files).

share|improve this question
    
Your doing this with Python? –  Dynamic Apr 22 '12 at 23:38
add comment

1 Answer

Many people speak highly of Solr. It specializes in searching large quantities of text, and should be usable from Python.

I suppose you could also do something custom with Hadoop. It directly supports map reduce operations, though it might be easier to work in Java than Python with Hadoop.

share|improve this answer
    
Solr is good... :) –  c0da May 7 '12 at 5:18
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.