If it is relatively easy to get good code coverage from profiling (because profiling tells you which functions are/aren't called, how many times, and with what parameters), how do I get good performance scenario coverage when doing performance benchmarks?
How do I even know that there isn't some "performance blackholes" which can't be discovered except when certain test parameters are very near the "blackhole"?
For a toy example, a sorting algorithm can be tested with data sets of size
1, 10, 100, 1000, 10000, .... An example of non-numerical coverage would be to test the sorting algorithm with
unsorted data, or
evil-constructed data intended to expose the worst case. These scenarios have been exhaustively investigated by the academia in the case of sorting algorithms.
How to apply that thinking into other kinds of software systems?