I currently work on a ~30 person development team that frequently releases a very large codebase of significant complexity, so some of our experiences may be of interest.
When it came time to setup our test approach, we initially split off a team of 4 people to setup the "harness", the core structure of the code supporting the automation of our tests. All tests are written in Cucumber, and the step definitions are relatively light, mostly making a couple calls into the harness that does the real work.
Once the harness was to the point that the team was able to write tests, about half of our entire group transition over to writing tests. The exact balance onbviously depends on the level of testing needed for your own project. Some things we learned early on:
- Create a couple tests and then loop back and harmonize them. Start building a standard so that new tests match the style of what is already in place. We ended up doing this later on, and had to go back and standardize what had been written so far.
- Code review feature files. Aggresively look for step definition duplication, when people go head down writing these things they start missing that someone already wrote the exact same thing.
- Make sure you are testing the right thing. For example, if submitting a form should do X, make sure the test is about submitting the form and not about clicking "Submit". The step definition may click the button, but the feature file shouldn't have to change if the name of a button does.
- If a test is likely to change because of a change in implementation without a change in functionality, then the test probably will break down the road. This mostly goes with 3.
We got to the point that running all of our tests from start to finish took an obscene amount of time. If you think you may get to the same point, think early about automating test environment setup so that you can easily run tests across multiple machines. We can deploy our entire production system to a clean host in under a minute, which allows us to scale up testing in burst very easily.
Whenever we prepare to do a release we cut a build, and then run it through these tests. We review any failures, and determine if they need to be fixed before delivery. If they do, fix and repeat. The entire process from first build to delivery usually takes under a week.
We have just recently gotten to the point that we are integrating together our Dev and QA teams. Now, any feature we are going to develop has a feature file written before anything else. Then we develop the code for the feature. And right after that the feature file is run. This only becomes possible once you have enough people on your team that understand BDD/your harness that they can help the rest of the team produce good tests.