(Upfront, I'm not a test or agile expert, just trying to push us to be better. If I'm wrong on anything I'd love your guy's opinions).
So I've been pushing for us to become a real agile shop and less of a "calling ad-hoc agile" shop. So I'm iteratively trying to improve our process. I've found that its been good to slowly build towards "doing it right": I can get buy in for small changes, or large changes on their own. So we've got slow iterations (about a month) that I've been using to slowly fold more and more in. My personal goal is to get us to a process that support continuous integration: at that point I think we can do things like speed up our iterations at much lower cost.
The next bit that I've sold management on that we need to improve is our testing. We've never had a real solid test strategy, so I'd like to start to build up our capabilities there. We've already introduced "Test Driven Development", using Google Test (unit test) framework, which has brought us very far. But these unit tests are not enough to test the system as a whole. We're supplementing this with our old way of testing, ad-hoc system and functional tests that are really specific to the project, that are rewritten every time, and that are not automated. To make matters worse, our functional and system tests are often more similar to characterization tasks than pass-fail tasks: our system/functional test tools need to be able to support advanced plotting and number crunching, which mean a lot of them are in Matlab shudder.
From what I can tell, a good automated test system has two parts: the Test Framework (something akin to the roboframework, ect) that figures out what tests to run, calls those tests, orgamizes the results, ect...the thing that actually does the automation. That uses the Test Tool(s), the fixture that actually performs your test. I may be viewing this wrong, but the Test Framework is like a huge batch file, and the Test Tools are the tests.
So I see two possible ways to go with "what to do next".
Implement (could be writing our own, though I'd prefer to acquire/modify) a Test Framework. Leave our ad-hoc system tests, but let the framework take over automation. The goal would to have a way to hit "Test" and have all the tests run.
Figure out a way that we can use to make Test Tools more efficiently and effectively.
I'm not sure what the best way to go is here, or if any of the following matter:
- We do both windows and linux development, though usually not at the same time
- While I want to focus on C++, C with CMake, we also do C# and Matlab development. The ideal framework would support all of these, even if it is different tools.
- The "testers", "test automators", and "developers" are all going to be pulled from the same group of highly-competent developers (for now.)
- I need to support command-line executable and libraries now, but would like to support GUIs (QT and .NET) in the future.
The scope of 2 seems HUGE, though doing 1 first seems like putting the cart before the horse. Do I have a semi-accurate picture of whats going on? I've read a few things that use Google Test as their test framework, but doesn't better fit the definition of test tool? Am I biting off more than I can chew?