Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm building and application that requires huge transactions/sec of data and I need to use IPC to for the mutithreaded mutliprocceses communication, I know that there are a lot of methods to be used but not sure which one to choose for this application.

This is what the application is gonna have, 4 processes, each process has 4 threads, the data chunk that needs to be transferred between two or more threads is around 400KB. I found that fifo is good choice except that it's 64K which is not that big so i'll need to modify and recompile the kernel but not sure if this is the right thing to do? Anyway, I'm open to any suggestions and I'd like to squeeze your experience in this :) and I appreciate it in advance.

share|improve this question
    
have a look - stackoverflow.com/q/4512967/1437962 –  Yusubov Jul 8 '12 at 16:10

2 Answers 2

up vote 1 down vote accepted

I maninly program for Windows but I think that the solutions are the same for Linux. I would use one of the following:

  • Named Pipes: Fast, have build in client server paradigm, work nice with multiple threads, eg. multiple readers / single writter etc...
  • Shared memory: Probably the fastest method of IPC, but you need to do your own synchronization, resource locking etc...
  • Sockets: Same as Pipes, little bit slower, but can communicate over machines or internet.

I would go for Named Pipes. I use them a lot and they are fast and reliable. Also if you need to move so much data per second (don't know exactly how much) them maybe you need to rethink your approach. Extremes often show bad design choices.

EDIT:

For names pipes here is a basic example.

share|improve this answer
    
do you think it's ok to use multiple fifo and read them sequentially? –  poly Jul 8 '12 at 16:59
1  
On Windows Pipes are in fact memory mapped files. You use then just the way you use files. On Linux I guess should be very similar or the same. You read and write from / to pipes in chunks the size of your choosing. So you can write / read 64K per chunk or you can do the whole content in one read / write. –  Runner Jul 8 '12 at 17:12
    
And yes you can read sequentially. –  Runner Jul 8 '12 at 17:22
    
I just checked it is the same as on Windows. I edited the answer with example. You use read() and write() to access data inside the pipe. –  Runner Jul 8 '12 at 17:28

If you're passing large amounts of memory around, you absolutely need shared memory. The moment you copy that 400k buffer you're taking a large perf hit.

So if you allocate the buffers in a shared memory area, you can just pass the reference/pointer/handle/whatever around and you're good. You will have to be careful about simultaneous writing to these buffers, but that's still much faster than passing a copy around. Its quite easy to do as well if you keep a 'in use' block at the beginning of the buffer too.

On Linux, shmget is the place to start.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.