I've been developing concurrent systems for several years now, and I have a pretty good grasp on the subject despite my lack of formal training (i.e. no degree). There's a few new languages that have become popular to at least talk about lately that are designed to make concurrency easier such as Erlang and Go. It appears that their approach to concurrency echoes my own experience as to how to make systems scalable and take advantage of multiple cores/processors/machines.
However, I find that there are very few tools to help visualize what you intend to do, and verify that you are at least close to your original vision. Debugging concurrent code can be a nightmare with languages that are not designed for concurrency (like C/C++, C#, Java, etc.). In particular, it can be near impossible to recreate conditions that happen readily on one system in your development environment.
So, what are your approaches to designing a system to deal with concurrency and parallel processing? Examples:
- How do you figure out what can be made concurrent vs. what has to be sequential?
- How do you reproduce error conditions and view what is happening as the application executes?
- How do you visualize the interactions between the different concurrent parts of the application?
I have my own answers for some of these, but I'd also like to learn a bit more.
So far we have a lot of good input. Many of the articles linked to are very good, and I've already read some of them.
My personal experience with concurrent programming leads me to believe you need a different mindset than you do with sequential programming. The mental divide is probably as wide as the difference between object oriented programming and procedural programming. I'd like this set of questions to focus more on the thought processes necessary (i.e. theory) to systematically approach the answers. When providing more concrete answers, it helps to provide an example--something you went through personally.
Goal for the Bounty
Don't tell me what I should do. I already have that under control. Tell me what you do. Tell me how you solve these problems.