Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Imagine a scenario as follows: Lets say you have a central computer which generates a lot of data. This data must go through some processing, which unfortunately takes longer than to generate. In order for the processing to catch up with real time, we plug in more slave computers.

Further, we must take into account the possibility of slaves dropping out of the network mid-job as well as additional slaves being added. The central computer should ensure that all jobs are finished to its satisfaction, and that jobs dropped by a slave are retasked to another.

The main question is: What approach should I use to achieve this?

But perhaps the following would help me arrive at an answer: Is there a name or design pattern to what I am trying to do?

What domain of knowledge do I need to achieve the goal of getting these computers to talk to each other? (eg. will a database, which I have some knowledge of, be enough or will this involve sockets, which I have yet to have knowledge of?)

Are there any examples of such a system? The main question is a bit general so it would be good to have a starting point/reference point.

Note I am assuming constraints of c++ and windows so solutions pointing in that direction would be appreciated.

share|improve this question
Look for queue or workflow systems, examples: aws.amazon.com/sqs en.wikipedia.org/wiki/Message_queue etcetera. The queue system can take care of distribution, checking and re-assigning. –  Luc Franken Jan 2 '13 at 11:03
The term you're looking for is parallelization. Before you head down this road, make sure your problem can actually be solved in parallel (e.g., the result for item n doesn't depend on the result for item n-1). –  Blrfl Jan 2 '13 at 11:27
Look at boinc –  MichaelT Jan 2 '13 at 14:30
@Blrfl I think parallelization is not usually associated with this, parallelization is usually associated with multi-threaded or multi-process execution on a single machine using multiple cores/processors or concurrent execution on the same machine. Distributed computing is what he's referring to and in his description it doesn't have guarantees of concurrency, just guarantees of asynchrony (which is to say non-deterministic execution which may be concurrent, synchronous, or delayed in queue) –  Jimmy Hoffa Jan 2 '13 at 15:21
I'm surprised nobody has mentioned hadoop. –  Kevin Jan 2 '13 at 16:19
show 3 more comments

1 Answer

Yes. This pattern is known as distributed computing(or distributed programming or whatever cool word you want to put after distributed). My suggestion will be not to build this in-house before looking at other solutions. You can look at this stack overflow question for various options. And then take calculated decision.

share|improve this answer
Don't forget the new favorite word for this: the cloud! or cloud computing. If you do it with your own computers people call them a personal cloud or mini cloud. Marketting nonsense, this has been known as you said as distributed computing for years, just tossing out there for the OP to know where he reads about cloud stuff it's talking about exactly this. –  Jimmy Hoffa Jan 2 '13 at 15:16
@JimmyHoffa: the word "cloud" is just as often used to indicate that your data as well as the software you use to work with it are on the internet somewhere instead of your local machine. And yet other times, it means that you're using a virtual machine that runs on a cluster rather than a single monolithic VM host; the obvious advantage being transparent zero-downtime failovers. –  tdammers Jan 3 '13 at 0:11
@tdammers true, but none the less the type of system described by the OP is one of the many utilizing the marketing moniker of the Cloud, along with as you pointed out practically everything else. That's what makes it a buzzword, it was invented without definition so people applied it to tons of different stuff so it effectively means everything and nothing. Hooray for buzzwords. –  Jimmy Hoffa Jan 3 '13 at 0:14
@JimmyHoffa: It's also what makes mentioning the word "cloud" here completely non-constructive. –  tdammers Jan 3 '13 at 0:21
It could make for a good word to use against management. "What is your solution to this data processing problem?" "Well, we could distribute it to a network of computers and compute them in parallel." BLANK LOOK. "We can build a mini Cloud." "OKAY CARRY ON" –  Morpork Jan 3 '13 at 1:03
add comment

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.