Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I need to process 2 million messages per second (perhaps in a scale out configuration) and route each message to a N delegates or multicast delegates.

Question

How should I structure my application in C# so that I can achieve the best performance when receiving the message, and routing it to the correct delegate?

Additional Details

Each inbound message has the following properties in the format of a JSON array:

  • Customer
  • Category
  • TopicID
  • DateTime
  • Data[]

Each message will be processed by a delagate function that is interested one or more of those properties.

Simple examples of delegate processing needed:

  • A customer counter function may count the quantity of messages per customer,
  • A category counter may count the quantity of messages that counter
  • A customer-counter will count messages unique to that customer.
share|improve this question
    
Are the delegates regularly changing, or are they merely there for future use? Are most messages going to most delegates? Regardless, this seems tailor made for something like Steam Insight where you can specify/compile standing queries to run across an event/input stream. –  Telastyn May 9 '12 at 15:53
    
@Telastyn What a great find! That seems to be exactly what I need.. –  makerofthings7 May 9 '12 at 15:54
    
StreamInsight can handle up to 10,000 events per second. not 1M –  Steven A. Lowe May 9 '12 at 15:55
    
I'm assuming I'll need to scale out regardless of my solution, and the aggregation of data will have a delay. I'm thinking of a MapReduce -ish type solution in addition to what comes up here. –  makerofthings7 May 9 '12 at 16:04
3  
@Steven: no single server will stand 200Gbps stream of data, it needs massive scale-out. –  vartec May 9 '12 at 16:06
show 3 more comments

1 Answer

200Gb of data per second?

To do this you're not going to get too much help with C#, all the High Performance Computing applications are written in C/C++, so for a start, you should be looking to go this direction simply to leverage their expertise, libraries and frameworks.

That said, a lot of telecoms work is done using functional languages, and they deliver input streams of a magnitude similar to what you're expecting, so try Haskel too.

In any case, your first task is to split that input stream to several servers, then pass them to more servers for processing. In these cases, you don't need delegates, you need a full message passing architecture that is stateless and memory efficient. Try OpenMPI for some hints.

share|improve this answer
1  
I'd recommend Erlang, a language built for concurrency. –  Falcon May 9 '12 at 18:22
    
d'oh - you know, I meant Erlang, just typed Haskel for some reason. –  gbjbaanb May 9 '12 at 23:05
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.