I need to handle and process ~5 million news articles, an amount small enough to fit on a commodity desktop but too large to process serially. Thus far, I have been using Python/
pickle for serialization, but this is easily getting out of hand. However, I do not have a server much less a cluster immediately available for distributed storage.
What tools can I use to store, query, and operate on this size of a dataset? The majority of operations I need to do can be formulated as Map operations; the only exception is that the Natural Language Processing portion which is handled by calling Stanford's CoreNLP software suite (it needs a few seconds to load some large parameter files).