What is the aim of software testing?

Question

Having read many books, there is a basic contradiction:

Some say, "the goal of testing is to find bugs" while other say "the goal of the testing is to equalize the quality of the product", meaning that bugs are its by-products.

I would also agree that if testing would be aimed primarily on a bug hunt, who would do the actual verification and actually provide the information that the software is ready?

Even e.g. Kaner changed his original definition of testing goal from bug hunting to quality assessment provision but I still cannot see the clear difference. I perceive both as equally important.

I can verify software by its specification to make sure it works and in that case, bugs found are just by products. But also I perform tests just to break things.

Also what definition is more accurate?

_{Note above I am primarily referring to software testing as a process.}

Answer 1

As I'm sure you are aware, there are many different types of software testing, such as unit testing, integration testing, acceptance testing, etc. So it's kind of an umbrella term for all of those, and at this very high level of discussion, we can only really talk about "quality", as a broad term. You are simply validating the software against whatever measurements of acceptability you wish to apply. At different levels and types of testing, these will vary greatly, and the only real common ground is the quality aspect.

Bugs (as traditionally defined) are a specific type of problem with the software, but there are many others. Unless we're discussing a specific, lower level of testing, it doesn't make sense to limit the definition to bugs. Is a UI which is a bit too slow a bug? What about the fact that we forgot to tell the developers to add a basket to our web store? Perhaps the confusion comes in with specific types of testing being referred to as "software testing", but it's really just semantics.

If pushed to formalise the definition, I would say that testing is a process of validating the software against your requirements (which can be qualitative, too). A bug is just a very specific violation of requirements (specifically, one which the developer thought already worked correctly).

EDIT: I should probably add that the word "bug" has very different meanings to different people, and we should actually start this semantic discussion by defining it. I'm using the definition from a developer's perspective - this doesn't work as I (the developer) intended it. It is typically based on either a very specific requirement, or a very specific interpretation of a requirement. The client's definition is typically similar - this doesn't work as I (the client) intended it, which is a very different thing indeed. Using the latter definition, you could almost equate quality and bugs, because anything that doesn't meet the client's wishes is a "bug".

Answer 2

From Daniel B's answer:

I'm using the definition from a developer's perspective - this doesn't work as I (the developer) intended it. It is typically based on either a very specific requirement, or a very specific interpretation of a requirement. The client's definition is typically similar - this doesn't work as I (the client) intended it, which is a very different thing indeed.

This is essentially the difference between verification and validation. Verification answers the question "Did we build it right?" Validation answers the question "Did we build the right thing?" Verification testing and validation testing are rather different things. Verification is a much easier task than validation. With verification, you know what to test against: the requirements (or stories) that spell out what the software is supposed to do. There's a problem here: What if those requirements or stories are wrong? How do you test that problem? That's what validation attempts to address.

Yet another component used in some circles is the concept of accreditation. This becomes important when software is reused. Example: Suppose you are building a simulation of a vehicle and need a good model of its inertial measurement unit. You find an existing IMU model in the components model library. This existing model has been thoroughly verified against requirements and validated against reality. The testing is very extensive, including comparisons against flight data. Verified and validated! Sounds good! Just reuse it as-is.

Then again, maybe not. The intended use of that model might have been quiescent operations, your use is to model a rocket during launch phase. The behavior of the IMU during launch will be close to spec behavior: in other words, lousy. IMUs typically perform much, much better than spec during quiescent operations. The intended use of that model does not match your intended use. You had better not reuse it. Accreditation attempts goes beyond verification and validation. It answers the question "Is this the right thing for this specific intended use?"

Another example is the first flight test of the Ariane 5 rocket. The software bug that led to the failure of flight 501 ranks as one of the most infamous and most expensive software bugs of all time. Flight software is extremely expensive to build. To avoid these huge costs, the Ariane 5 flight software reused big chunks of the Ariane 4 flight software. Extensively verified and validated, and already used in a real-world setting. So just reuse it as-is. Wrong. It should have been accredited for reuse. A supposedly "can't happen" event involving a 64 bit to 16 bit conversion overflow happened to happen, and the vehicle failed as a result.

Answer 3

Identify software regressions as soon as they present themselves.

Unit Testing, in particular, is meant to identify regressions early in the building/testing/deploying chain

Acceptance Testing is more on the lines of fullfilling a contract with a client. But then again, if one part of an acceptance test doesn't pass while it was instead supposed to, you have identified a regression to handle.

Answer 4

What is the aim of software testing?

In short: As question authors comment says "in general, software testing as a process." - Your question is broad, and here is its definition in Wikipedia article.

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation.

Thus, aim of software testing is to provide independent information about the quality of the product/software. - How it needs to be done and sub-process of software testing? - is a different question to look for.

Edit: Software testing process needs to be provided independently based on business requirements. Otherwise, there would be less value in it. In fact, big scope software projects (like national real estate projects or similar) do have a separate biding for quality control, testing and software verification/acceptance processes.

Answer 5

software testing is not aimed at finding bugs, it's aimed at preventing bugs. Proper testing in early stages prevents bugs making it into production systems.

Answer 6

I believe the book, "The Art of Software Testing" by Glenford J. Myers defines it best:

"Testing is the process of executing a program with the intent of finding errors."

He contrasts this definition with several common definitions:

• "Testing is the process of demonstrating that errors are not present."
• "The purpose of testing is to show that a program performs its intended functions correctly."
• "Testing is the process of establishing confidence that a program does what it is suppose to do."

Rather than trying to prove that a program works, we should assume the program has errors, and the goal of software testing is to find them. In doing so, the quality of the software is raised, which is the ultimate aim of software testing.

Answer 7

I think investments in testing are taken to "improve the user experience".

Lots of times bugs are left in because they don't adversely affect the use of the product. Likewise, "equalizing the quality of the product" will also be broken down by how useful it is to work on a particular area. Lastly, the "good enough to ship" criteria must be recognized in that the end state for testing is always subjective.

Answer 8

A software with a great quality is, among other things, a software with 0 (or very few) bugs. So a bug hunting is a process to improve the quality.

A software which meets all its features is also a great quality software, so testing to validate the features is a process to improve the quality.

A software which a good user experience is a great quality software, and so on...

Well, I think the aim of sotware testing is to improve the quality, then you can do some bug hunting tests, some validation and verification of features, some tests on UI, etc.