In a project where there are non-functional requirements which specify the maximum execution time for a specific action, QA must check the performance of this action on a dedicated machine using precise hardware under precise load, both hardware and load being specified in the requirements.
On the other hand, some erroneous changes to the source code may severely impact the performance. Noticing this negative impact early, before the source code reaches source control and is verified by QA department, could be beneficial in terms of time lost by QA department reporting the issue, and by the developer fixing it several commits later.
To do this, is it a good idea:
To use unit tests to have an idea of the time spent executing the same action² n times,
To use per-test timeout through
[TestMethod, Timeout(200)]attribute in C#?
I expect several problems with this approach:
Conceptually, unit tests are not really for that: they are expected to test a small part of a code, nothing more: neither the check of a functional requirement, nor an integration test, nor a performance test.
Does unit test timeout in Visual Studio measure really what is expected to be measured, taking in account that initialization and cleanup are nonexistent for those tests or are too short to affect the results?
Measuring performance this way is ugly. Running a benchmark on any machine¹ independently of the hardware, load, etc. is like doing a benchmark that shows that one database product is always faster than another. On the other hand, I don't expect those unit tests to be a definitive result, nor something which is used by the QA department. Those unit tests will be used just to give a general idea about the expected performance, and essentially to alert the developer that his last modification broke something, severely affecting performance.
TDD is impossible for those tests. How would it fail, in the first place, before starting to implement code?
Too many performance tests will affect the time required to run the tests, so this approach is limited to short actions only.
Taking in account those problems, I still find it interesting to use such unit tests if combined with the real performance metrics by QA department.
Am I wrong? Are there other problems which makes it totally unacceptable to use unit tests for this?
If I'm wrong, what is the correct way to alert the developer that a change in source code severely affected performance, before the source code reaches source control and is verified by QA department?
¹ Actually, the unit tests are expected to run only on developer PCs having comparable hardware performance, which reduces the gap between the fastest machines which will never be able to fail the performance test, and the slowest machines which will never succeed at passing it.
² By action, I mean a rather short piece of code which spends a few milliseconds to run.