Sign up ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free.

OK, so, background:

I have a little "server" app/module that is written in C++. This is kindof a tunnel application that provides a standardized API via TCP/IP[a] and on the other hand talks to the "real" server via TCP/IP[a] via it's own interface.

It doesn't even officially support multiple clients, but due to the inherently multithreaded nature of the processing that is done by the server and by the client (plus asynchronous callbacks) the tunneling app itself is multithreaded and runs pretty well during normal operations.

The application uses a number of simple mutexes (Win32 CRITICAL_SECTIONS) to lock down the code paths that would create problems when executed concurrently and this does work pretty well in practice.

A little picture:

[proprietary server]        [tunneling app]        [client]
                     <--1-                  <--1-
                              (upcalls)    |<--2-
                     <--3-                  <--3-
                                           | -2-->
                      --1->    (returns)     -1-->
                      --3->                  -3-->
                      -a-->    (callbacks)   -a1->
                      -b--> |                -a2->

So pretty much everything goes with regards to call chains.

We have now identified a number of problems in fringe cases like calls during initialization or shutdown of the server and/or tunneling app and I'm struggling to properly fix these as they are mostly related to multithreaded object and thread lifetimes and just throwing more mutexes or more locks at the problem really doesn't cut it.

As for the Question:

What I am looking for is advice and guideline on how to write multithreaded (C++) code that would work in all corner cases from the start and doesn't involve just locking everything down. Possibly with a specific focus on the context of heavy multithreading in a network communication scenario.

Any insights welcome.


[a] : It's a CORBA communication interface. But The problems I see are really not related to the actual messaging mechanism used, as the messaging part on the wire works very well.

share|improve this question

2 Answers 2

Running multiple threads in a shared memory space and trying to figure out where to put locks does not work, as you discovered. You need to approach the problem from the opposite end: threads really should share nothing and communicate with each other via messages. See this article on Erlang-style concurrency.

share|improve this answer
Upvoted. Maybe I should do some re-reading on Erlang. Then again, I'm not convinced it's that simple in the average C++ context (performance/memory/design constraints). Doesn't share nothing eventually lead to copy-everything? Have you ever applied this to a C++ project? – Martin Ba Aug 4 '11 at 14:19
@Martin: It is a matter of design really. For instance, if multiple threads need to access a file, one approach would be to put a lock around each piece of code that touches that file, but it is error-prone and does not scale. Another approach would be to have a specialized thread that has exclusive access to a file, runs in a loop, takes requests from a queue and puts results in another queue. All other threads would now submit requests to this specialized thread instead of competing for the lock. Of course, this is not a real-life example (there is async IO for that) but the strategy is. – Nemanja Trifunovic Aug 4 '11 at 14:28
@Martin - I've done things this way in C++. The way we did it was, not to "copy state", but rather "grant state" during IPC. That is, if a second thread needed to be able to manipulate state, the current owning thread needed to give ownership of that state to the other thread for manipulation - and the old owner would then no longer be able to manipulate that state, unless it got granted back to it. – Aidan Cully Aug 4 '11 at 14:31

throwing more locks does cut it you know :) Eventually you'll be running your multithreaded app in a single-threaded mode (ie only 1 at a time) but it'll run :)

I think what you need is a mechanism for communicating the state of the app to the runnign threads. If you crash when shtting down its because some threads still expect the app to be present, even though its closing down its data. structures. In this case, you need to stop accepting new threads, but wait for the old ones to terminate. Alternatively, you need to send a messagwe to each running thread to tell it that's its time to stop whatever it was doing. Which way you choose depends on the app.

You shouldn't have any problem on initialisation - just prevent any threads from being created until you're ready for them.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.