We build some software platform, and now we need to make hardware sizing sheet for the server-side piece of this software. So, I should prepare some mathematical model that, for example, given the number of concurrent users, can predict the number of physical servers or storage capacity or network bandwidth. How to make such model?
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
You may want to start with a mathematical model, but I'd advise instead to do a simulation - make a "user": a program that acts like a user and generate randomized user-like activity. Then, load the system with a real-life like dataset (sized accordingly) and deploy your users against the structure you created. Rinse and repeat until you get as many measures for every data point so that you have a reasonable confidence the number represents the performance at a given load and that noise has been reduced to a level that doesn't interfere with your measurements. Keep an eye on maximums, because they may appear in real-world usage and derail the application. Also, remember to test it with absurd numbers of users and absurd volumes of data, as it often offers you some insight on problems that may not be evident on more normal scenarios.
If possible, publish alongside your chart the methodology you used (how your users behave, what hardware did you use - if real metal, if EC2 instances - and how many measurements you made).
This is a highly worthy exercise, BTW. You may discover some bottlenecks you never imagined you'd have, as well as new ways to tune your application on different hardware configurations that never occurred to you.
I come from a web background and I try to make every system I build capable of generating one or more JMeter configuration files for load testing purposes. Nobody knows how to better simulate load than the system itself - it has access to previous usage patterns, as well as to its own working database.
And then when you have a ton of experimental data, you can refine your mathematical models in order to predict conditions you may not easily simulate.
I think the advice to prototype is right on. Run it, see how it runs with X users. Then, add in a "fudge factor" on top of your results so that you have some breathing room with your customers. So if your test results find that you need 512 MB of RAM for 100 users, write 768 MB or 1 GB in your recommended specs.
The big guys (HP, Microsoft, Oracle) all have recommendations. Routinely those recommendations are either loosely adhered to in development and test environments, or ignored completely because the hardware used is "whatever is available". Just a thought about that.
A "purely mathematical" approach is unlikely to be very accurate. One of the basic questions to ask when optimising software for performance is "have you measured it"?
My advice would be to perform some stress testing of your software. Do this in a carefully controlled way, and observe the behaviour of the system while it's under different levels of stress. This information, combined with your inside knowledge of the system itself, should allow you to come up with some educated guesses about the way your system will perform with different hardware and different loads. I would advise against trying to extrapolate to hardware that varies wildly from your test setup - it's one thing to predict what will happen if you go from two cores to four, but will that scale to 64 cores, or make efficient use of hundreds of GB or RAM? I would stay away from wild guesses like that.