Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I was thinking about experimenting a bit with using mapreduce and such on a dataset we have to see if we can benefit from it in any way.

What route should I go if I want to be able to instantly, without any major modification, deploy my findings to some kind of grid/cloud-system to see if there is any extra benefit to have it scaled in the cloud.

share|improve this question

"The Cloud" is just a bunch of machines that you don't own. Running on those machines is no different than running on the machines that you do own, except that you pay for the time and bandwidth.

If you're using a tool like Hadoop, adding machines is just a matter of configuration (you don't mention platform, but it's the main player in the non-Google map-reduce space, which is just one of the big-data techniques).

My recommendation is that you work through the tutorial, get a single-node implementation of your application running, then immediately move to a cloud service like Amazon EC2. You can start with two small instances for about a dollar a day, and increase the number of machines from there.

As a cautionary note: unless you're dealing with 100s of gigabytes of data (and preferably terabytes), I don't think you'll see much benefit of either horizontal scaling or "the cloud." There is a not-insignificant time to setup and teardown map-reduce operations.

share|improve this answer
Why immediately? Didn't you just say that it doesn't matter? – Robert Harvey Jun 4 '13 at 17:37
Um, a cloud is a cloud whether or not you own the machines in question. To me, the distinction comes when you add a layer on top of the network of machines that essentially behaves like an operating system for groups of computers. But everyone has their own definition. – btilly Jun 5 '13 at 5:59
@RobertHarvey I agree with the "immediately" bit. When you have one machine, it is too easy to write to a file in one place, and expect that file to exist in another. Once you've got 2 machines, you notice that you've introduced those dependencies because things don't work. But if the software scales horizontally, there should be no real difference between 5 and 500 machines other than that the second is faster. (Note I said "if" and the if in question is always false to some extent...) – btilly Jun 5 '13 at 6:01

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.