Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We process messages through a variety of services (one message will touch probably 9 services before it's done, each doing a specific IO-related function). Right now we have a combination of the worst-case (XML data contract serialization) and best-case (in-memory MSMQ) for performance.

The nature of the message means that our serialized data ends up about 12-15 kilobytes, and we process about 4 million messages per week. Persistent messages in MSMQ were too slow for us, and as the data grows we are feeling the pressure from MSMQ's memory-mapped files. The server is at 16GB of memory usage and growing, just for queueing. Performance also suffers when the memory usage is high, as the machine starts swapping. We're already doing the MSMQ self-cleanup behavior.

I feel like there's a part we're doing wrong here. I tried using RavenDB to persist the messages and just queueing an identifier, but the performance there was very slow (1000 messages per minute, at best). I'm not sure if that's a result of using the development version or what, but we definitely need a higher throughput[1]. The concept worked very well in theory but performance was not up to the task.

The usage pattern has one service acting as a router, which does all reads. The other services will attach information based on their 3rd party hook, and forward back to the router. Most objects are touched 9-12 times, although about 10% are forced to loop around in this system for awhile until the 3rd parties respond appropriately. The services right now account for this and have appropriate sleeping behaviors, as we utilize the priority field of the message for this reason.

So, my question, is what is an ideal stack for message passing between discrete-but-LAN'ed machines in a C#/Windows environment? I would normally start with BinaryFormatter instead of XML serialization, but that's a rabbit hole if a better way is to offload serialization to a document store. Hence, my question.

[1]: The nature of our business means the sooner we process messages, the more money we make. We've empirically proven that processing a message later in the week means we are less likely to make that money. While performance of "1000 per minute" sounds plenty fast, we really need that number upwards of 10k/minute. Just because I'm giving numbers in messages per week doesn't mean we have a whole week to process those messages.

=============== edit:

Additional information

Based on the comments, I'll add some clarification:

  • I'm not sure serialization is our bottleneck. I've benchmarked the application, and while serialization does show up in the heat graph, it's only responsible for maybe 2.5-3% of the service's CPU utilization.

  • I'm mostly concerned about the permanence of our messages and potential misuse of MSMQ. We are using non-transactional, non-persistent messages so we can keep queueing performance up, and I would really like to have at least persistent messages so they survive a reboot.

  • Adding more RAM is a stopgap measure. The machine has already gone from 4GB -> 16GB of RAM and it's getting harder and harder to take it down to continue adding more.

  • Because of the star routing pattern of the application, half the time an object is popped then pushed to a queue it doesn't change at all. This lends itself again (IMO) to storing it in some kind of key-value store elsewhere and simply passing message identifiers.

  • The star routing pattern is integral to the application and will not change. We can't application-centipede it because every piece along the way operates asynchronously (in a polling fashion) and we want to centralize the retry behavior in one place.

  • The application logic is written in C#, the objects are immutable POCOs, the target deployment environment is Windows Server 2012, and we are allowed to stand up additional machines if a particular piece of software is only supported in Linux.

  • My goals are to maintain current throughput while reducing memory footprint and increasing fault tolerance with a minimum outlay of capital.

share|improve this question
    
Comments were cleaned up as the relevant points were incorporated into the question. –  ChrisF Oct 22 '13 at 21:06
    
It would make sense to address the most pressing problem before worrying about swapping queuing subsystems (though that may still be worth doing eventually). The fact that memory is growing out of control suggests that there are still leaks somewhere. What (if any) memory profiling has been done? –  Dan Lyons Oct 25 '13 at 19:10
    
@DanLyons: the only memory growth is in MSMQ. Nobody really talks about it, but it seems to be because of non-persistent messages which are all memory-mapped. Since we're serializing a lot of data, it does keep a substantial amount of memory allocated. The memory is (eventually) reclaimed as the messages are consumed and MSMQ's internal cleanup runs. –  insta Oct 25 '13 at 20:06

3 Answers 3

up vote 1 down vote accepted

Here are some queue benchmarks you might be interested in. MSMQ should be capable of handling 10K messages per second. Could it be a configuration issue or perhaps the clients aren't keeping up with reading the queue? Also note how blazingly fast ZeroMQ is in those benchmarks(around 100K messages per second), it doesn't offer a persistence option but it should get you to where you want to be performance wise.

share|improve this answer

We had a somewhat similar situation several years ago, with a queued message system (audio fingerprints in our case). We strongly valued persistency of the enqueued data packets, but we found out that enqueuing everything to disk and consuming the queue from disk was very expensive.

If we switched to memory-based queues, performance was exceptional, but we had a major problem. Every once in a while the consumers of the queues became unavailable for a considerable amount of time (the consumer and producer elements in our case are connected via WAN), so the producer's queue would grow to a point it became unmanageable and like your case, once memory consumption was very high, excessive memory thrashing during swapping brought the system to a complete crawl.

We designed a queue we christened VMQueue (for Virtual Memory Queue, a very bad name in retrospective). The idea of this queue is that if the consumer process is running up to par, in other words, processing fast enough to be able to keep the number of enqueued elements below a certain level, then it has basically the same performance of a memory-based queue. However, when the consumer slows down or becomes unavailable and the producer queue grows to a certain size, then the queue will begin automatically paging elements to-and-from disk (using BinaryFormatter serialization by the way). This process keeps memory usage completely controlled, and the paging process is fast, or at least much faster than the virtual memory swapping occurring during heavy-memory load. Once the consumer manages to drain the queue below the threshold, it resumes working as a pure memory-based queue

If the system crashes or reboots, then the queue is able to recover all paged elements that were stored to disk, it will only lose the elements that were still kept in memory before the crash. If you can afford losing a limited number of packets during a crash or reboot, this queue might be helpful.

If you are interested, I can share the VMQueue class source code so you can play around with it. The queue will accept any class that is marked as Serializable. Upon creation of the queue you establish the size of the page in number of elements. The class interface is virtually the same of a standard Queue class. The code is very old however, (.net 1.1) so no generic interface exists unfortunately.

I know that moving from the proven MSMQ technology is a huge bet, however this queue has been working reliably for almost 6 years and has allowed us to survive and recover from scenarios where the producer machine has been offline for several weeks! Please let me know if you are interested. :)

share|improve this answer

HP ProLiant ML350G5 system gets 82k transactions per minute - ie it has over 8x that "10k/minute" throughput you mentioned.

Performance: 82,774 tpmC

Also, to be honest, I'd have just gone with 64 or even 128 GB of RAM - RAM is cheap. Greenspun points out the difference between "throw RAM at it" and "get a smart MIT-educated guy to optimize it", and the RAM wins.

He ended up with a SQL Server machine equipped with 64 GB of RAM and a handful of front-end machines running ASP.NET pages... The site, swaptree.com, handles its current membership of more than 400,000 users (growing rapidly) without difficulty...

Note "the machine has already gone to 16 GB of RAM" is far from enough, with an article pointing out a server that was handling 400k users on 64 GB of RAM.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.