8

Having read many books, there is a basic contradiction:

Some say, "the goal of testing is to find bugs" while other say "the goal of the testing is to equalize the quality of the product", meaning that bugs are its by-products.

I would also agree that if testing would be aimed primarily on a bug hunt, who would do the actual verification and actually provide the information that the software is ready?

Even e.g. Kaner changed his original definition of testing goal from bug hunting to quality assessment provision but I still cannot see the clear difference. I perceive both as equally important.

I can verify software by its specification to make sure it works and in that case, bugs found are just by products. But also I perform tests just to break things.

Also what definition is more accurate?

Note above I am primarily referring to software testing as a process.

2
  • 1
    What sort of testing are you primarily referring to? Oct 5, 2012 at 8:54
  • In general, software testing as a process.
    – John V
    Oct 5, 2012 at 8:55

9 Answers 9

19

As I'm sure you are aware, there are many different types of software testing, such as unit testing, integration testing, acceptance testing, etc. So it's kind of an umbrella term for all of those, and at this very high level of discussion, we can only really talk about "quality", as a broad term. You are simply validating the software against whatever measurements of acceptability you wish to apply. At different levels and types of testing, these will vary greatly, and the only real common ground is the quality aspect.

Bugs (as traditionally defined) are a specific type of problem with the software, but there are many others. Unless we're discussing a specific, lower level of testing, it doesn't make sense to limit the definition to bugs. Is a UI which is a bit too slow a bug? What about the fact that we forgot to tell the developers to add a basket to our web store? Perhaps the confusion comes in with specific types of testing being referred to as "software testing", but it's really just semantics.

If pushed to formalise the definition, I would say that testing is a process of validating the software against your requirements (which can be qualitative, too). A bug is just a very specific violation of requirements (specifically, one which the developer thought already worked correctly).

EDIT: I should probably add that the word "bug" has very different meanings to different people, and we should actually start this semantic discussion by defining it. I'm using the definition from a developer's perspective - this doesn't work as I (the developer) intended it. It is typically based on either a very specific requirement, or a very specific interpretation of a requirement. The client's definition is typically similar - this doesn't work as I (the client) intended it, which is a very different thing indeed. Using the latter definition, you could almost equate quality and bugs, because anything that doesn't meet the client's wishes is a "bug".

4
  • I think your last paragraph sums it up real nice.
    – harald
    Oct 5, 2012 at 9:07
  • +1. I meant to vote it up at the time that I elaborated upon it. Apparently I forgot to do so. Oct 7, 2012 at 3:54
  • Is a UI which is a bit too slow a bug? - could I call it Usability Bug? What about the fact that we forgot to tell the developers to add a basket to our web store? Could I call it bug in the requirements?
    – Tarun
    Feb 12, 2014 at 3:57
  • @Tarun you could definitely go that route. The word bug is very easily misunderstood though (usually along the lines of "programmer messed up"), so perhaps it's not the best terminology. Regarding the "UI too slow" issue, I was leaning towards a qualitative measure which is often not specified, yet implicit and expected by clients. In this case, it could almost be both a "usability bug" and a "requirements bug".
    – Daniel B
    Feb 12, 2014 at 6:23
10

From Daniel B's answer:

I'm using the definition from a developer's perspective - this doesn't work as I (the developer) intended it. It is typically based on either a very specific requirement, or a very specific interpretation of a requirement. The client's definition is typically similar - this doesn't work as I (the client) intended it, which is a very different thing indeed.

This is essentially the difference between verification and validation. Verification answers the question "Did we build it right?" Validation answers the question "Did we build the right thing?" Verification testing and validation testing are rather different things. Verification is a much easier task than validation. With verification, you know what to test against: the requirements (or stories) that spell out what the software is supposed to do. There's a problem here: What if those requirements or stories are wrong? How do you test that problem? That's what validation attempts to address.

Yet another component used in some circles is the concept of accreditation. This becomes important when software is reused. Example: Suppose you are building a simulation of a vehicle and need a good model of its inertial measurement unit. You find an existing IMU model in the components model library. This existing model has been thoroughly verified against requirements and validated against reality. The testing is very extensive, including comparisons against flight data. Verified and validated! Sounds good! Just reuse it as-is.

Then again, maybe not. The intended use of that model might have been quiescent operations, your use is to model a rocket during launch phase. The behavior of the IMU during launch will be close to spec behavior: in other words, lousy. IMUs typically perform much, much better than spec during quiescent operations. The intended use of that model does not match your intended use. You had better not reuse it. Accreditation attempts goes beyond verification and validation. It answers the question "Is this the right thing for this specific intended use?"

Another example is the first flight test of the Ariane 5 rocket. The software bug that led to the failure of flight 501 ranks as one of the most infamous and most expensive software bugs of all time. Flight software is extremely expensive to build. To avoid these huge costs, the Ariane 5 flight software reused big chunks of the Ariane 4 flight software. Extensively verified and validated, and already used in a real-world setting. So just reuse it as-is. Wrong. It should have been accredited for reuse. A supposedly "can't happen" event involving a 64 bit to 16 bit conversion overflow happened to happen, and the vehicle failed as a result.

1
  • +1 for the elaboration of verification vs. validation, and I tend to agree that validation is often the harder of the two.
    – Daniel B
    Oct 7, 2012 at 7:57
5

What is the aim of software testing?

In short: As question authors comment says "in general, software testing as a process." - Your question is broad, and here is its definition in Wikipedia article.

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation.

Thus, aim of software testing is to provide independent information about the quality of the product/software. - How it needs to be done and sub-process of software testing? - is a different question to look for.

Edit: Software testing process needs to be provided independently based on business requirements. Otherwise, there would be less value in it. In fact, big scope software projects (like national real estate projects or similar) do have a separate biding for quality control, testing and software verification/acceptance processes.

3
  • Independent? Hardly ever. Most tests are written by the same organization that creates the product. With test driven development, it's not just the same organization, it's the same people. Independent verification and validation is expensive and is rarely used outside of the realm of highly critical systems. Oct 7, 2012 at 4:00
  • 1
    -1 for citing wikipedia. It's a too broad definition and, because taken out of wikipeda, just someone elses opinion.
    – Andy
    Nov 5, 2012 at 12:32
  • 2
    @Andy, if quote says the source that is not the reason for down-voting. Secondly, Wikipedia is public ally available for editing and improvements. Thus it is NOT a personal opinion. It is community opinion.
    – Yusubov
    Jan 9, 2013 at 12:26
4

Identify software regressions as soon as they present themselves.

Unit Testing, in particular, is meant to identify regressions early in the building/testing/deploying chain

Acceptance Testing is more on the lines of fullfilling a contract with a client. But then again, if one part of an acceptance test doesn't pass while it was instead supposed to, you have identified a regression to handle.

1
  • The way I see the term “regression testing” used, the latter would not be a regression, since the feature never worked that way. Regression testing is one of the things I as a developer am heavily interested in, but it is not the whole of software testing, you also need verification and validation, as David Hammen nicely explained the terms in his answer. Feb 8, 2014 at 1:33
2

I believe the book, "The Art of Software Testing" by Glenford J. Myers defines it best:

"Testing is the process of executing a program with the intent of finding errors."

He contrasts this definition with several common definitions:

  • "Testing is the process of demonstrating that errors are not present."
  • "The purpose of testing is to show that a program performs its intended functions correctly."
  • "Testing is the process of establishing confidence that a program does what it is suppose to do."

Rather than trying to prove that a program works, we should assume the program has errors, and the goal of software testing is to find them. In doing so, the quality of the software is raised, which is the ultimate aim of software testing.

2
  • 1
    but we should take care that testing is not assuring quality.
    – alinoz
    Nov 5, 2012 at 10:18
  • As we used to say in the chip test business, "You cannot test quality into a product"!
    – Bill IV
    Feb 7, 2014 at 2:06
1

@David Hammen's answer is very well put, though not exactly what I'd have said. I agree that Verification answers "Did we built this one correctly?". Anything produced by a process can be Verified. Manufacturing involves constant verification that the thing being produced has been produced correctly.

He then defines Validation, which we agree is different, as "Did we build the right thing?" I would say that Validation moves in the opposite direction, to exhaustively confirming precisely correct functioning of a design. More like "Objectively demonstrate that the solution is correctly designed". The right grades of bolts, the right sizes of internal variables. The pieces are up to the job.

David's Validate, "Did we build the right thing?" is a high-level question that isn't something you can run against the daily build, thumbs up or thumbs down. Its a judgement of the requirements and to a lesser extent, the design. Its not a sensible question addressed to a text box on a screen or a parameter in an API. Not sure what the one-word name is for requirement correctness, maybe Requirement Validation. Exhaustively proving that the requirements correspond to the needs of the end user.

By contrast, my definition for Validate is proof of correctness of a design, objective tests that show the pieces selected will do the job. The Ariane IV software that was unsuitable for Ariane V would fail here, because Ariane IV had a limited range of angle-rate changes. The code was optimized for this limited range, and Ariane V was capable of a larger range of angle rates, which caused the overflow. When both onboard computers crashed on overflow, and did it again after auto-reboot, the destruct system was triggered.

As Dykstra said, "Premature optimization is the root of all evil."

In my definitions, the requirements are presumed to be correct and accepted, validated by Requirement testing. Design or Code Validation proves that the design, perhaps a bit of the implementation, is correct. It still has to be executed correctly, but confirming that is Verification, testing based on accepted Requirements and an accepted Design.

You'll note that this hews painfully close to the Waterfall model of development, which seems harmful if believed to describe complex systems. None the less, Requirements are different than Design and Code is a third thing entirely. I guess my plea is that the elements in the Waterfall are useful descriptions, but that 'complete' is misleading, so I've changed it to 'accepted' which suggests contingency and mutability.

0

software testing is not aimed at finding bugs, it's aimed at preventing bugs. Proper testing in early stages prevents bugs making it into production systems.

-1

I think investments in testing are taken to "improve the user experience".

Lots of times bugs are left in because they don't adversely affect the use of the product. Likewise, "equalizing the quality of the product" will also be broken down by how useful it is to work on a particular area. Lastly, the "good enough to ship" criteria must be recognized in that the end state for testing is always subjective.

-1

A software with a great quality is, among other things, a software with 0 (or very few) bugs. So a bug hunting is a process to improve the quality.

A software which meets all its features is also a great quality software, so testing to validate the features is a process to improve the quality.

A software which a good user experience is a great quality software, and so on...

Well, I think the aim of sotware testing is to improve the quality, then you can do some bug hunting tests, some validation and verification of features, some tests on UI, etc.

Not the answer you're looking for? Browse other questions tagged or ask your own question.