A bit of a background here - we are a small team (of 5) of RAD developers responsible for internal software development in a big non-software company. "Internal software" varies from a desktop .NET application using MSSQL server as a backend to Python scripts running on the background to MS Word documents and templates - a zoo of technologies.
The whole team consist of all-arounders able to get the requirements from users, code it up, test it and deploy into production. Once the software in the production it is being looked after by another team but it usually easy for us to intervene if something goes wrong.
All sounds good so far, but there is a problem - being a RAD team we have to release often, and there is no day going by without us releasing new versions of one or two applications(or it could be a script, updated word document, C++ console app, etc) into the production. We do a development testing and also do involve end-users by letting them run the software in UAT environment ...
... but the bugs are creeping in to production anyway. Users do understand that these bugs and the occasional instability is the price they are paying for getting what they want really quickly, but at the same time it got us thinking - perhaps we could improve our development or a release practices to improve the stability of the software and reduce the number of bugs we introduce when adding a new functionality.
The good thing - we don't really have much of the processes in the first place, so it should be easy to start improving, the bad thing - being a small RAD team we don't really have the much time and resources to implement something big, but we have been thinking about the following initiatives and would welcome any feedback, tips, hints and suggestions.
Currently some of the applications are being released into the production straight after the developer testing, bypassing the user acceptance testing. That practice should be discontinued and even a small change has to be tested by a end-user. Each application will have a dedicated beta-tester selected from the end-users. Only after a beta-tester has ok-ed the new release it is promoted from test to production environment.
We don't conduct code reviews - but we'll start doing code reviews before one of us checkin the changeset. I was also thinking about a "rollout review" - basically one of the developers has to sit next with the other watch him/her doing the software rollout (copy binaries, update configs, add new table to database, etc) - it usually only takes a 5-10 minutes so it won't take much of a "rollout review" time.
How to miminise the rollback time when a new release is proven to be buggy enough to be pullout from production and to be replaced by a good previous version. We do store a history of all releases (as a binaries) to make it easy to go one version back - and though it is quick "overwrite a newly released binaries with a previous versions binaries" it is still a manual process which is error prone and demanding at times "what if the rollback will fail and will render the system unusable instead of buggy".
This is where we ran out of our ideas and we'd like to get your feedback on these and if you could share some simple release / dev process improvement advices - that would be awesome.