Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

My datasource sends information in 100 batches of 100 records with a delay of 1 to 3 seconds between batches.

I would like to start processing data as soon as it's received, but I'm not sure how to best approach this.

Some ideas I've been playing with include:

  • yield
  • Concurrent Dictionary
  • ConcurrentDictionary with INotifyProperyChanged
  • Events
  • etc.

As you can see I'm all over the place, and would appreciate some tested guidance on how to approach this

share|improve this question

2 Answers 2

up vote 6 down vote accepted

I would look into the TPL Dataflow APIs for this kind of work. At bare minimum you would do something as simple as:

ActionBlock<MyData> myActionBlock = new ActionBlock<MyData>(data =>
{
    // your processing logic here
});

And then as your messages arrive from whatever external event you just post them to that block:

myActionBlock.Post(data);

That's it. TPL Dataflow then implements all the producer/consumer scale for you by farming the work you've provided in your action definition out to as many possible worker threads as the machine can handle (which is based on all the usual TPL heuristics).

From there you can start to get a little more advanced by simply taking control of governing exactly how much capacity/scale you want with ExecutionDataflowBlockOptions. You can also hand in a CancelationToken via EDBO.

Then you can start to get really advanced by using the composition features of TPL Dataflow's many different "block" types and do things like introduce a BufferBlock which, in your case, would allow you to receive that burst of 100 items and not block the producer who is doing the .Post call. Only if you reach 100 would the producer then be blocked from adding any more. There's also BatchBlock which, depending on your exact workload, it might make more sense to use so that you can send maybe 20 items off to a worker thread to be processed rather than 1 at a time.

You can link together any concoction of blocks as you can imagine. There's filtering, transforming... you name it. To top it all off you can even write your own custom blocks. For example, if you wanted to introduce temporal coupling into the mix you could, for example, batch up to 20 item or if you haven't received 20 for 10s you send through however many you have already received. The possibilities are pretty much limitless.

share|improve this answer
    
I hate when I write a big library to solve a lot of problems only to find its already built into .NET. +1 –  insta Jan 30 '13 at 15:09

get back to the basics with a producer-consumer buffer (look at thread safe queues)

  • your receiving code pushes each record to a queue

  • then several processing threads will pull a record and process it in a loop

share|improve this answer
    
Perhaps expose concurrentQueue and have a separate thread poll that queue, or maybe I can fire a delegate or event everytime I get new data to be processed. –  makerofthings7 Oct 18 '12 at 23:25
    
if you want multiple threads doing the processing the queue is the better solution –  ratchet freak Oct 18 '12 at 23:28
    
Sticking with .NET naming what would you name the classes? xDatasource, yAdapter, zModel? (substitute the first letter with a properly capitalized prefix... where Datasource exposes the queue, and Adapter populates the model?) –  makerofthings7 Oct 19 '12 at 11:53
1  
@makerofthings7 - Its entirely up to you there is no incorrect answer. –  Ramhound Oct 19 '12 at 14:00

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.