Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I have a question, it's about why programmers seems to love concurrency and multi-threaded programs in general.

I'm considering 2 main approaches here:

  • an async approach basically based on signals, or just an async approach as called by many papers and languages like the new C# 5.0 for example, and a "companion thread" that manages the policy of your pipeline
  • a concurrent approach or multi-threading approach

I will just say that I'm thinking about the hardware here and the worst case scenario, and I have tested this 2 paradigms myself, the async paradigm is a winner at the point that I don't get why people 90% of the time talk about multi-threading when they want to speed up things or make a good use of their resources.

I have tested multi-threaded programs and async program on an old machine with an Intel quad-core that doesn't offer a memory controller inside the CPU, the memory is managed entirely by the motherboard, well in this case performances are horrible with a multi-threaded application, even a relatively low number of threads like 3-4-5 can be a problem, the application is unresponsive and is just slow and unpleasant.

A good async approach is, on the other hand, probably not faster but it's not worst either, my application just waits for the result and doesn't hangs, it's responsive and there is a much better scaling going on.

I have also discovered that a context change in the threading world it's not that cheap in real world scenario, it's in fact quite expensive especially when you have more than 2 threads that need to cycle and swap among each other to be computed.

On modern CPUs the situation it's not really that different, the memory controller it's integrated but my point is that an x86 CPUs is basically a serial machine and the memory controller works the same way as with the old machine with an external memory controller on the motherboard. The context switch is still a relevant cost in my application and the fact that the memory controller it's integrated or that the newer CPU have more than 2 core it's not bargain for me.

For what i have experienced the concurrent approach is good in theory but not that good in practice, with the memory model imposed by the hardware, it's hard to make a good use of this paradigm, also it introduces a lot of issues ranging from the use of my data structures to the join of multiple threads.

Also both paradigms do not offer any security abut when the task or the job will be done in a certain point in time, making them really similar from a functional point of view.

According to the X86 memory model, why the majority of people suggest to use concurrency with C++ and not just an async approach ? Also why not considering the worst case scenario of a computer where the context switch is probably more expensive than the computation itself ?

share|improve this question

migrated from stackoverflow.com Dec 14 '12 at 23:16

This question came from our site for professional and enthusiast programmers.

2  
One way to compare would be to look at the JavaScript world, were there is no threading and everything is aggressively asynchronous, using callbacks. It works, but it has its own issues. –  Steven Burnap Dec 15 '12 at 0:13
1  
@StevenBurnap What do you call web workers? –  user16764 Dec 15 '12 at 0:19
    
"even a relatively low number of threads like 3-4-5 can be a problem, the application is unresponsive and is just slow and unpleasant." => It might be due to poor design / inappropriate use of threads. You typically find that kind of situation when your threads keep exchanging data, in which case multi threading might not be the right answer or you might need to re-partition the data. –  assylias Dec 15 '12 at 0:25
1  
@assylias Feel free to substitute 'UI thread' with 'main thread', it doesn't matter. Excessive locking without understanding the effect can easily cancel out any benefits gained from multi-threading. Especially nested thread locks. –  Evan Plaice Dec 15 '12 at 8:38
2  
You say "programmers seems to love concurrency and multi-threaded programs in general" I doubt that. I would say "programmers hate it" ... but often it's the only useful thing to do ... –  johannes Dec 15 '12 at 11:33
show 2 more comments

3 Answers

up vote 14 down vote accepted

You have multiple cores/procesors, use them

Async is best for doing heavy IO bound processing but what about heavy CPU bound processing?

The problem arises when single-threaded code gets stuck on a long-running process. For instance, remember back when printing a word processor document required the program to lock up until the request was finished being sent? In such a case, the easiest way to keep the main thread responsive is to push the long running parts onto a background thread.

In practice, async and multi-threaded/process code each have their uses.

You can see the trend in the major cloud platforms, as they will offer instances specialized for CPU bound processing and instances specialized for IO bound processing.

Examples:

  • Storage (ex Amazon S3, Google Cloud Drive) is CPU bound
  • Web Servers are IO bound (Amazon EC2, Google App Engine)
  • Databases are both, CPU bound for writes/indexing and IO bound for reads

To put it into perspective...

A webserver is a perfect example of a platform that is strongly IO bound. A multi-threaded webserver that assigns one thread per connection doesn't scale well because every thread incurs more overhead (ie memory). Whereas an IO bound webserver would use a single address space. Apache's scalability woes come from the fact that it is implemented using a multi-process/multi-threaded model.

Likewise, an application specialized for encoding video would work much better in a multi-threaded environment because the heavy processing involved would lock the main thread until the work was done. There are ways to mitigate this but it's much easier to have a single thread managing a queue, a second thread managing cleanup, and a pool of threads managing the heavy processing. Communication between threads happens only when tasks are assigned/completed so thread-locking overhead is kept to a bare minimum.

The best application often uses a combination of both. A webapp, for instance may use nginx (ie async single-threaded) as a load balancer to manage the torrent of incoming requests, a similar async webserver (ex Node.js) to handle http requests, and a set of multi-threaded servers handle uploading/streaming/encoding content, etc...

There have been a lot of religious wars over the years between multi-threaded, multi-process, and async models. As with the most things the best answer really should be, "it depends."

It follows a the same line of thinking that justifies using GPU and CPU architectures in parallel. Two specialized systems running in concert can have a much greater improvement than a single monolithic approach.

Neither are better because both have their uses. Use the best tool for the job.

share|improve this answer
    
why a programmer should go multi-process ? I mean I assume that with more than 1 process you also need some kind of inter-process communication that can add a significant overhead, is this something like the old windows-programmer way of doing things ? when should i go multi-process ? Thanks for your reply by the way, really good picture of what async and multi-threaded are good for. –  user1849534 Dec 15 '12 at 16:36
    
You are assuming that the interprocess communication would increase the overall overhead. However if the processing state is immutable, or only needs to handle synchroniztion upon start/completion. it can be far more efficient to fan-out into more parallel tasks. The actor pattern is a good example, and if you haven't read about it - it is really worth reading up on. akka.io –  sylvanaar Dec 15 '12 at 22:01
    
@user1849534 Multiple threads can talk to each other via shared memory + locking or IPC. Locking is easier but harder to debug if you make a mistake (ex missed a lock, dead lock). IPC is best if you a have a lot of worker threads because locking doesn't scale well. Either way, if you're using a multi-threaded approach it's important to keep communication/synchronization across threads to an absolute minimum (ie to minimize overhead). –  Evan Plaice Dec 15 '12 at 22:53
    
@akka.io You're completely right. Immutability is one way to minimize/eliminate the overhead of locking but you still incur the time cost of context switching. If you'd like to extend the answer to include the details about how immutability can resolve thread synchronization issues, feel free. The main point I was aiming to illustrate is, there are cases where async communication has a distinct advantage over multi-threaded/process and vice versa. –  Evan Plaice Dec 15 '12 at 23:03
    
(cont) But, honestly if I needed a lot of CPU-bound processing capability I'd skip the actor model and built it to capable of scaling to multiple networking nodes. The best solution I have seen for this is using 0MQ's task ventilator model over socket-level communications. See Fig 5 @ zguide.zeromq.org/page:all. –  Evan Plaice Dec 15 '12 at 23:11
show 2 more comments

Microsoft's asynchronous approach is a good substitue for the most common of the purposes for multithreaded programming: improving responsiveness with respect to IO tasks.

However, it's important to realize that the asynchronous approach is not capable of improving performance at all, or improving responsiveness with respect to CPU intensive tasks.

Multithreading for Responsiveness

Multithreading for responsiveness is the traditional way to keep a program responsive during heavy IO tasks or heavy computation tasks. You save files on a background thread, so that the user can continue their work, without having to wait for the hard drive to finish its task. The IO thread often blocks waiting for some portion of a write to finish, so context switches are frequent.

Similarly, when performing a complex calculation, you want to allow regular context switching so the UI can remain responsive, and the user doesn't think the program has crashed.

The goal here is not, in general, to get the multiple threads to run on different CPUs. Instead, we're just interested in getting context switches to happen between the long-running background task and the UI, so that the UI is able to update and respond to the user while the background task is running. In general, the UI won't take up much CPU power, and the threading framework or OS will usually decide to run them on the same CPU.

We actually lose overall performance due to the extra cost of context switching, but we don't care because performance of the CPU wasn't our goal. We know that we usually have more CPU power than we need, and so our goal with regard to multithreading is to get a task done for the user without wasting the user's time.

The "Asynchronous" Alternative

The "asynchronous approach" changes this picture by enabling context switches within a single thread. This guarantees that all of our tasks will run on a single CPU, and may provide some modest performance improvements in terms of less thread creation/cleanup and fewer real context switches between threads.

Instead of creating a new thread to await the receipt of a network resource (e.g. downloading an image), an async method is used, which awaits the image becoming available, and, in the meantime, yields to the calling method.

The main advantage here is that you don't have to worry about threading issues like avoiding deadlock, as you aren't using locks and synchronization at all, and there's a bit less work for the programmer setting up the background thread, and getting back on the UI thread when the result comes back in order to update the UI safely.

I haven't looked too deeply into the technical details, but my impression is that managing the download with occasional light CPU activity becomes a task not for a separate thread, but rather something more like a task on the UI event queue, and when the download completes, the asynchronous method is resumed from that event queue. In other words, await means something akin to "check whether the result I need is available, if not, put me back in this thread's task queue".

Note that this approach would not solve the problem of a CPU-intensive task: there's no data to await, so we can't get the context switches we need to happen without creating an actual background worker thread. Of course, it might still be convenient to use an asynchronous method to start the background thread and return the result, in a program that pervasively uses the asynchronous approach.

Multithreading for Performance

Since you talk about "performance", I'd also like to discuss how multithreading can be used for performance gains, something that's entirely impossible with the single-threaded asynchronous approach.

When you're actually in a situation where you don't have enough CPU power on a single CPU, and want to use multithreading for performance, it's actually often difficult to do. On the other hand, if one CPU isn't enough processing power, it's also often the only solution that could enable your program to do what you'd like to accomplish in a reasonable timeframe, which is what makes the work worthwhile.

Trivial Parallelism

Of course, sometimes it can be easy to get real speedup from multithreading.

If you happen to have a large number of independent computation-intensive tasks (that is, tasks whose input and output data are very small with respect to the calculations that must be performed to determine the result), then you can often get significant speedup by creating a pool of threads (sized appropriately based on the number of available CPUs), and having a master thread distribute the work and collect the results.

Practical Multithreading for Performance

I don't want to put myself forward as too much of an expert, but my impression is that, in general, most practical multithreading for performance that happens these days is looking for places in an application that have trivial parallelism, and using multiple threads to reap the benefits.

As with any optimization, it's usually better to optimize after you've profiled your program's performance, and identified the hot spots: it's easy to slow down a program by deciding arbitrarily that this part should run in one thread and that part in another, without first determining whether both parts are taking up a significant portion of CPU time.

An extra thread means more setup/teardown costs, and either more context switches or more inter-CPU communication costs. If it's not doing enough work to make up for those costs if on a separate CPU, and doesn't need to be a separate thread for responsiveness reasons, it will slow things down for no benefit.

Look for tasks that have few interdependencies, and that are taking up a significant portion of the runtime of your program.

If they have no interdependencies, then it's a case of trivial parallelism, you can easily set up each with a thread and enjoy the benefits.

If you can find tasks with limited interdependence, so that locking and synchronization to exchange information doesn't slow them down significantly, then multithreading can give some speedup, provided you're careful to avoid the dangers of deadlock due to faulty logic when synchronizing or incorrect results due to not synchronizing when it's necessary.

Alternatively, some of the more common applications for multithreading aren't (in a sense) looking for speedup of a predetermined algorithm, but instead for a larger budget for the algorithm they're planning to write: if you're writing a game engine, and your AI has to make a decision within your frame rate, you can often give your AI a bigger CPU cycle budget if you can give it its own CPU.

However, be sure to profile the threads and ensure that they're doing enough work to make up for the cost at some point.

Parallel Algorithms

There are also a lot of problems that can be sped up using multiple processors, but that are too monolithic to simply split between CPUs.

Parallel algorithms have to be carefully analyzed for their big-O runtimes with respect to the best available non-parallel algorithm, as it's very easy for the inter-CPU communication cost to eliminate any benefits from using multiple CPUs. In general, they must use less inter-CPU communication (in big-O terms) than they use calculations on each CPU.

At the moment, it's still largely a space for academic research, in part because of the complex analysis required, in part because trivial parallelism is quite common, in part because we don't yet have so many CPU cores on our computers that problems which can't be solved in a reasonable time frame on one CPU could be solved in a reasonable time frame using all of our CPUs.

share|improve this answer
add comment

the application is unresponsive and is just slow and unpleasant.

And there's your problem. A responsive UI does not make a performant application. Often the opposite. A bunch of time is spent checking UI input rather than having the worker threads do their job.

As far as 'just' having an async approach, that's multithreading as well although tweaked for that one particular use case in most environments. In others, that async is done via coroutines that are... not always concurrent.

Frankly, I find async operations to be more difficult to reason about and use in a way that actually provides benefit (performance, robustness, maintainability) even compared to... more manual approaches.

share|improve this answer
    
why ? for example what you find so bananas in the boost signals2 library ? –  user1849534 Dec 15 '12 at 16:26
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.