Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

If it is relatively easy to get good code coverage from profiling (because profiling tells you which functions are/aren't called, how many times, and with what parameters), how do I get good performance scenario coverage when doing performance benchmarks?

How do I even know that there isn't some "performance blackholes" which can't be discovered except when certain test parameters are very near the "blackhole"?

For a toy example, a sorting algorithm can be tested with data sets of size 1, 10, 100, 1000, 10000, .... An example of non-numerical coverage would be to test the sorting algorithm with sorted data, unsorted data, or evil-constructed data intended to expose the worst case. These scenarios have been exhaustively investigated by the academia in the case of sorting algorithms.

How to apply that thinking into other kinds of software systems?

share|improve this question

1 Answer 1

Ask the users for scenarios in which your application is "too slow". Performance can be only analyzed relative to users' needs. The question is never "is it fast?" but always "is it fast ENOUGH?". Therefore, without involving the users, you can't do any reasonable performance testing. They will tell you what you should be benchmarking and how fast is fast enough.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.