how to deal with parallel programming

Question

I know that parallel programming is a big resource in computer graphics, with moder machines, and mayebe a computing model that will be grow up in the near future (is this trend true?).

I want to know what is the best way to deal with it. there is some practical general purpose usefulness in studying processor n-dimensional mesh, or bitonic sort in p-ram machines or it's only theory for domain specific hardware used in real particular signal elaborations of scientific simulations?

Is this the best way to acquire the know how for how to become acquainted with cuda or opencl? (i'm interested in computer graphics applications)

and why functional programming is so important to understand parallel computing?

ps: as someone has advice me i have forked this discussion from http://stackoverflow.com/questions/4908677/how-to-deal-with-parallel-programming

Answer 1

I came to it from a supercomputing background (generally for scientific and engineering uses). The two main styles of parallelism are shared memory, where one program runs with multiple threads in the same address space -- a frequent way to implement that is OPENMP -- and message passing, with MPI being the most popular software.

If you only care about utilizing a few processors, say the number you can get on a dual socket machine, which supports 2 chips, I'd start with OPENMP. You usually use this by adding directives (pragmas) to for loops. Sometimes you can take an existing application and in a piecemeal fashion parallelize it, one loop at a time.

Message passing, with MPI or some other package, is usually more difficult, although you are guaranteed that the difference processes have independent memory spaces. But, whenever data has to be shared, it must be buffered, and sent out to one or more other processes, which must make calls to receive the data. Message passing parallelism, usually requires that the application be designed from the ground up for parallelism. But given a sufficiently parallel application the number of processors that it can scale to is essentially unlimited. Clusters of hundreds to thousands of cores are not uncommon.

You can also mix methods, with each MPI process, utilizing several processors to compute its own locally parallelized piece of a larger parallel application.

I haven't used OpenCL, but isn't it a way to use the graphics chip to perform calculations? Cuda is currently used to program what are being called GPGPUs (General Purpose Graphical Processing Units), such as are provided by the NVidia Fermi.

Personally, I think efforts to learn and write Cuda are likely to be wasted. I think that the numbers of true general purpose (x86) cores per chip on offer, will continue to rise, and the number of floating point units per core (accessible via AVX) will also grow several fold in the next few years. So IMHO, using GPGPUs to make up for a dearth of floating point units per chip, will probably only a preferred short term solution. There may be some good jobs available for good Cuda programmers, just be prepared for the whole field to change drastically within a few years.

Answer 2

First about the functional languages, is that they only help because it's functions are inherently reentrant (since they don't modify any data outside them), and therefore can be run as a thread (maybe in a different processor) more easily than a regular procedural function (or object oriented method), but you can also archive that in a regular language, so you only need to learn functional languages if you have a real interest in them. (My guess is that you don't, and that you just learned that Lua and other functional languages are used a lot in game development, and you went with the tide)

Second, you don't need to learn OpenCL in order to do parallel computing, simple native threads, and other mechanisms are enough for that, because parallel computing can be used in games, if you have one thread that does all the drawing, and other thread that calculates the collisions, character paths, etc. in the background.

Answer 3

Starting with your last question first, functional programming isn't necessarily important to understanding parallel computing, but it can be useful for parallel programming. In particular, the lack of state modifications means that it's usually relatively easy to have some function x and function y execute in isolation from each other, and remain certain that they both work the same way as they would if they (for example) shared an address space. It also means that all communication between functions is easy to detect and generally well known from the beginning.

Continuing backward, I'd say the best way to understand something like Cuda or OpenCL is to study it directly. Most of the theoretical models assume that (for example) a CREW PRAM machine will use exactly that model at all times, under all circumstances. OpenCL, by contrast, allows you to specify what data is global and what data is local. Its memory model is described in pretty decent detail in the OpenCL specification. Cuda is slightly different of course but the same generally applies to it as well -- if you want to use Cuda, you're generally best off studying it directly. Yes, it wouldn't hurt to have also studied some of the more theoretical models first to get at least some idea of how basic algorithms get implemented in massively parallel machines, but don't expect most of it to apply directly.

I don't think there is one best way to deal with parallel programming, any more than there's on best way to deal with serial programming. There are (at least) half a dozen models/paradigms/whatever that serial programming has followed over the years. I'm pretty sure all of those and more besides will apply to parallel programming.

Answer 4

Parallel and distributed computing go hand-in-hand. Omega Centauri rightly pointed out the shared memory architecture. But shared nothing architecture based on the Actor Model seems to be the forerunner for general purpose computing. If you want to learn parallel programming in general, since CPU's are also multiplying their cores, I suggest you to learn the following

1. Distributed Computing in general. Distributed Systems: Concept and Design gives you an excellent insight
2. MapReduce - just to think in the parallel, distributed way
3. Erlang - Actual programming

Also, OpenCL is platform-independent while CUDA is not but CUDA is more mature. And a final note, neither MapReduce is a silver bullet nor Erlang is a speed machine