I recently asked this question and this comment and made me ponder the whole process of releasing an update to the customers. The next update for our software is planned and the previous time it went horribly wrong because of regression, bad testing, etc...
This time we want to prevent the scenario that we test via our customer and made some unit tests and other minor improvements. However, there is not enough time to make unit tests for all the features that lack unit tests or the features that need to be build yet. So the answer for this question 'Make more unit tests' is not fitting because of the lack of time, infrastructure etc...
Every bug and feature will be tested/reviewed by a different developer, but that won't be done perfectly of course (we're still human :)).
My question is: Once we released the update, how do we prevent the vicious circle of testing via customers while not be able to add more unit tests?
The previous time we tried our best to test the software before and after release. After the deadline we were still searching and fixing bugs but the customer would also find some of these. So we post an update as soon as we were confident that the patch was increasing quality. We were so busy with finding bugs and fixing them that we weren't spending any time on creating unit tests.
This time we don't have a solid plan yet, hence the question. As stated before we did added some unit tests, but surely not enough. We are also looking for new colleagues but they are not easy to come by and management has rather high standards. We do have quite some candidates and it looks like we do get at least one new colleagues very soon, but it still takes time to get him/her up and running. The first thing we put the new colleague on will be creating unit tests. So the dependencies there are experience, skills and the time he/she can join the time.
Another problem that comes in hand with the lack of developers is that we are so focused on getting the bugs and features fixed that we don't have much time to spend on thinking about 'how we can improve the update process'. (I ask these questions in my own spare time).
Summary of steps I will try to take:
- Try to get a tester before the release is done
- Add unit tests while creating the features, if there isn't enough time, schedule them for after the release.
- After releasing, if there is any regression, create unit tests when fixing the bugs.
- Release more often in smaller chunks after the release, making it easier to release features and repair bugs with a minor priority.