I was wondering how big companies of software developers check for bugs in their programs.
Do they just test it on several computers?
This question came from our site for professional and enthusiast programmers.
Here are some of the techniques that Google uses.
I have ranked these in what I suspect is descending order of effectiveness in catching bugs.
|show 2 more comments|
Bigger companies usually have whole entire Q/A departments which are responsible for testing code and making sure it works the way it is supposed to. Its usually just as you described-- a bunch of people testing lots of machines. Sometimes the tests are automated, sometimes they aren't. See Quality Assurance - Wikipedia
Many times, the developers themselves will find bugs during the development process. Also, customers are frequently the first to find a bug.
Smaller companies, like one I am currently working for, use the Agile Testing practice
I would say its about the maturity of a company and not the size :) There are large companies that have poor development practices and small companies that are on the bleeding edge.
In general a mature development team will engage in the following activities to 1; minimize introducing new bugs to the system and 2; find bugs in the existing system.
Unit testing: These are 'mini drivers' for individual methods to ensure that a method does what it says it does. These are always automated tests.
Integration testing: These tests aim to check that a larger unit of functionality works within the system. This might involve testing database integration or integration with third party libraries. These are automated tests as well.
Acceptance testing: Acceptance tests are written to test user requirements. These usually just test the 'happy path'. In my team, these tests are designed to show that if the user uses the functionality as it was designed to be used, they will have no trouble. Can be manual or automated.
Functional testing: These tests are similar to acceptance tests, but they also test the 'unhappy path'. These tests mean to test the not so obvious scenarios. Can be manual or automated.
Regression testing: We use this term to do a 'full testing' of the system before its released to the customers. Manual or automated.
Gorilla testing: (Manual only). This is the kind of testing when very smart humans intentionally try to break the application.
Performance testing Aims to make sure that performance is acceptable and does not degrade over time. Usually automated.
Stability testing: These tests are designed to make sure the system remains stable over time. Automated.
Continuous Integration: This is a system that automatically checks out your code, compiles it and runs your automated tests. Your faster tests (unit, integration) will run each time a dev commits code. Some other ones run nightly (acceptance, functional) or weekly (performance, stability).
Code coverage reports: Shows you how much of your code is tested. Code that has no test coverage is more likely to break.
Different tools that analyze the code: These usually show where the code needs to be re-factored to make it less prone to potential bugs.
Pair programming: Two developers working together to deliver functionality. "A cohesive pair is better than the sum of its parts."
The most important to take away is: automation and continuous integration.
It depends on the company and on the products it develops.
First, many companies enforce coding practices like code reviews and mandatory linting (automated bug detection tools) to reduce the amount of errors going in to the repository. Many companies also adopted unit testing. This is the case where I work (Google). When code is checked in, the tests are run against everything, to make sure no new errors are introduced.
Second, many companies have QA departments that are responsible for validating behavior. This is particularly common in Finance (where mistakes can be expensive and validation rules are complex), but also exists in companies that sell products to users where recalls can be expensive (e.g., Intel, Microsoft, etc.).
Third, whenever possible companies do Dogfooding (have their own users use the product internally) and then release limited betas. Many errors are caught at this stage. For example, folks working at Microsoft use newer internal versions of Office and Windows and DevStudio than what you have outside. Then limited groups of users or contracted companies get to sample it. Similarly, at Google we use internal versions of GMail and Docs prior to release. Game companies organize open betas to test their products and the load on the servers,etc.
Of course the answer is "It dpends", but I will give a sample from my largest project so far, which had at peak time around 50 developers involved.
The basic setup: A backend software for processing large amounts of data with BizTalk.
The first line of defense are the unit tests. In our case these got executed daily for everything checked into source control and usually some of them were executed manually by the developer before check-in. The unit tests were mainly written by the developers but sometimes amended with additional tests by the testers.
Next step was a weekly Virtual PC build, where the testers ran a series of mainly automated end-to-end tests on the data-flow based on the specification documents for each component.
After that the same Virtual PC was enriched with some business data quite close to the real thing and tested again with some specific use cases.
Then the Virtual PC was put together with other system components (also mostly virtual) from other departments to an integration test cycle based on end-to-end testing from the user entering the data to the end of the data flow.
On another track the installation packets were tested by the systems provider to see if they installed correctly on a production-like environment and if they could be rolled back if something failed.
After the installation on the production-like environment we ran load and stress tests there to test the overall stability (not something to be taken lightly when you run on 10 BizTalk servers, 8 SQL Servers and a bunch of other specialized hardware like an XML accelerator and a dedicated Archive - all clustered of course).
When we were satisfied with all the tests, the code was put into production. You get a pretty big latency to fix bugs in the code (like 4-6 weeks for the whole test cycle), and it's expensive to do all these tests, but the overall stability was pretty good. In fact the best I have seen so far. Again that's quite important on a system that processes several million dollars worth each day. Your requirements may vary, but that's how we have done it and it worked.
The original question seems more conceptually generic, than most of the highly detailed answers that were provided.
Let´s look it from a higher level (less detailed). Software is developed to attend to specific needs from someone (person, company, whatever).
Those needs need to be mapped into the individual stories/requirements that would be lately (in a construction phase) be implemented in source code.
Having the stories/requirements well defined is essential for the Quality Assurance (QA) team (the actual software testers) to validate the final code, when executed, attends to the demands of those stories and requirements. So for that purposee, the QA team creates "testcases" to do that validation.
The code, once released to the QA team, will then be tested and bugs will be identified. Bugs of differente types and severities. Those bugs are tracked and developers get them assigned to finally get them fixed.
The usage of virtual machines, nowadays, allows for one tester to run different environments in one same real hardware. But sometimes you end up needing some computers dedicated to the QA phase.
I hope that helps you understanding (roughly) the overall process.
Well, I hate to be cynical, but with the number of open bugs in a certain 'device' operating system it seems that the bigger and richer the company, the more bugs they are capable of creating and delivering to the end user. If the software kinda works and looks cool then they just release it anyway. If the managers think its ready, then its ready. That's when the really nasty bugs start coming out of the woodwork, and the end users get to be guinea pigs. Ten or so years later, most of the bugs will have been worked out (and a few added for good measure) and the company will be ready to move on to the next big idea.