Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

In the Software testing book from Koirala, Sheihk, they say:

A positive test is when you put in a valid input and expect some action to be completed in accordance with the specification.
A negative test is when you put in an invalid input and receive errors.

Part of my question:

Let's say there is a requirement: If the customer name=="James Bond", do something.

Is name<>"James Bond" negative or still positive test, verifying the requirement?

Or a login function:
Is correct login positive test case and incorrect negative?
Or Login failed is just confirmation of requirement that a password must be correct ->confirming invalid pass will not work.

For example, if the program is supposed to give an error when the person types in "101" on a field that should be between 1 and 100, then that is positive then if an error shows up. If, however, the application does not give an error when user types 101, then you have a negative test.

share|improve this question
3  
A spec should specify the behavior for all cases. If it does not say what has to happen when X==5 or X < 5 then the bavior is not defined and it does not make sense to test that case. –  Thomas Mar 23 '13 at 17:03
1  
A test with name<>"James Bond" will be a positive test, because such names are also valid (unless specified otherwise in the requirements). –  Rogério Mar 28 '13 at 19:39
add comment

9 Answers

up vote 3 down vote accepted
+50

Let me try to resolve your confusion in simple words.

Negative Test is a test designed to determine the response of the system outside of what is defined. So any kind of input which is not as per the design or requirement of the system is a part of negative test and your system should be capable enough to handle it. it should not crash or throw an unexpected error.

Thanks & I hope its simple and clear.

share|improve this answer
    
Well, it is not that clear as you see the other answers. What about that James Bond example. When user enters name that equals "James Bond", secret menu appears. Also negative test would be any other name and not opening the menu? –  user970696 Apr 1 '13 at 19:49
    
Hi Friend. New have a different look on the same scenario. Your Design Says that when user name is "James Bond" then Secret Menu Appears and if it is "not James Bond" then some other menu or no menu appears right? So in this situation either you will provide name "James Bond" or any other junk value and try to validate the requirement of your system. These both are still positive but if i will modify the system requirement by this "Only user id James Bond can log-in to system" then entering James Bond is Positive while providing any other input is Negative. Hope this will help you. –  Logicalj Apr 2 '13 at 5:01
    
"it should not crash or throw an unexpected error" - I disagree, in many cases crashing or throwing an error is exactly what should happen. –  Robert Johnson Apr 3 '13 at 11:44
2  
@Robert Johnson: not really. E.g. entering letters into numeric field should not just be allowed or ignored. One would not expect a crash or exceptions. Negative testing is mainly about robustness and thus you do not want to see crashes and errors, just invalid input properly handled. –  user970696 Apr 3 '13 at 11:48
    
Sure, in that example an error is not the right option, but there are plenty of cases where it would be. For example, what if the database that an application relies upon is inaccessible or corrupted? In that case the best thing you could do may be to display an error to the user, in which case your negative test would be to make sure that a "nice" error message is presented as opposed to, say, dumping out a stack trace to the screen. –  Robert Johnson Apr 3 '13 at 12:20
add comment

Positive and negative is one nomenclature, true and false another. In general, you have to distinguish two things, which are both of a boolean nature:

  1. Is the test itself written according to the requirements, or violating them? Let me refer to these as true and false from now on.
  2. Is the test run successful or not? Let me refer to these as positive and negative from now on.

What your quote denotes as "positive" and "negative" are in fact only two of the possible four combinations. The quoted phrase "positive" means a true-positive result, i.e. valid input and a successful test result. Accordingly, "negative" means false-negative, i.e. invalid input and a failed test.

Result classifications

Here's a brief description for each of the four possible combinations and what is hidden behind them:

True-positive

The simplest case, in which your input is adhering to the specification and the result is as expected. Not much to say about these really.

True-negative

These are good, too. You have provided valid input, and the failing test tells you that there is something wrong in your product.

False-positive

These are monsters. They show up in your reports as green checkmarks, but really what the test is testing is not adhering to the specification.

False-negative

Again you are testing something which doesn't make sense, and you do it in a way that fails. The good point about these is that you at least become aware of the problem by a failing test.

Application to 101 Example

Let's apply this to your 101 example:

True tests

First, you write two test cases: Test #1 checks that for valid inputs, the user does not see an error message, whereas Test #2 checks that for input outside of the valid range an error is shown. Both tests adhere to the specification, hence, you can only end up with the true-cases below:

  • Test#1 True-positive: I enter 50 and no error shows up
  • Test#2 True-positive: I enter 101 and an error shows up
  • Test#1 True-negative: I enter 50, but an error shows up
  • Test#2 True-negative: I enter 101, but no error shows up

The negative cases both indicate that you have to fix your implementations.

False tests

Now consider two more tests: Test #3 checks that when you input a number between 1 and 100, an error message shows up. And Test #4 checks that no error message is shown when you enter 101. Both of these tests are not adhering to the specification! But you can still execute them, because they are just executable tests after all. Here are the possible cases:

  • Test #3 False-positive: I enter 50 and an error shows up
  • Test #4 False-positive: I enter 101 and no error shows up
  • Test #3 False-negative: I enter 50, but no error shows up
  • Test #4 False-negative: I enter 101, but an error shows up

As you can see, due to the nature of these tests you will never get a True-X result. The false-negative results are nicer, because they point you to the problem.

Impact on development

Our problem as developers is that we only get to see the result of test executions, for example in an IDE, as success or failure, which maps to the latter positive or negative. You do not get any indication of whether your test is true or false (because it is far from trivial to automatically determine that a test adheres to a specification).

True-negatives are relatively simple to fix: you just fix your code to do what the customer wants it to do.

False-negatives are more trouble: As it is really the test itself, which is at fault, you risk losing valuable time trying to turn a False-negative into a False-positive by "fixing" your product code.

False-positives are not a problem at the time they occur, but further down the road. Imagine that all your tests succeed, but several versions later that one tests suddenly fails. It has been working all the time, so you will assume that some recent code change broke it, but in fact, the test itself was already broken.

You might argue that the False-X variants of tests should not be created - and of course that's right. But they usually do not come into existence, because someone writes them, but rather, because someone wrote them long ago and now someone else has changed the requirements/specification. A simple example would be to change your example to allow 51-150 as valid input. A test that checks for an error for the input 200 remains a True-X test, but the test that tries to enter 101 and expects an error becomes a False-X. Gladly, it will fail.

More problematic is the test that enters 50 and expects no error. Due to the changed specification, 50 is no longer a valid input, but you will not find out that your application still accepts it as valid easily, because your test still says everything is alright. The specification change changed the test from a True-positive into a False-positive.

Lesson to be learned: when changing your specification make sure to update your tests as well.

References

The nomenclature of true/false-positive/negatives comes from statistics, especially, medical tests. They often have the problem, that tests are not as deterministic in their nature as it is the case for software, so they are also interested in sensitivity and specificity.

Another nomenclature (also in statistics) is to refer to the false-X cases as type I and type II errors.

share|improve this answer
    
Thank you, yet this is not clear with the login or string example. More than results it is about whether it can be percieve as a negative test cases (login example). –  user970696 Mar 28 '13 at 15:30
add comment

I do not think it is a negative test, because the requirement dictates to show error when number ==101. Also typing 101 I am validating the requirement.

If you're testing the requirement that the program produces an error for input of 101, then entering 101 and expecting an error is a positive test.

If you're testing that the program accepts values between 1 and 100, then entering 101 and expecting an error is a negative test.

In other words, the difference between positive and negative test depends on exactly what you're testing. A positive test determines that the behavior works correctly (or doesn't), and a negative test determines that erroneous input fails.

share|improve this answer
1  
A test always verifies something that is in the requirements. If something is not a requirement, then it shouldn't have a test. In this example, we have two requirements: 1) the system must accept a number between 1 and 100 as valid and process it accordingly; and 2) the system must validate numbers outside that range as invalid and display an error instead. The first requirement calls for a positive test, while the second one for a negative test. –  Rogério Mar 28 '13 at 19:56
1  
@Rogerio Hardly. If the requirement is the field must accept only numbers, then you do not need to explicitly say it must not accept anything else. The original requirements already specified the group of valid inout. –  user970696 Mar 29 '13 at 7:20
    
@user970696 Yes, you don't need to say that it must not accept anything else, but you do need to say what should happen in that case. And that is a separate requirement; otherwise, you would have incomplete requirements and undefined behavior. –  Rogério Apr 1 '13 at 11:54
add comment

To me, the difference between positive and negative tests is simple.

A positive test will verify that valid input is correctly processed. A negative test will verify that invalid input causes some validation to fail. (In a positive test, all validations will necessarily pass - if one doesn't, the test fails.)

Or in other words: if all validations are expected to pass for the given input/scenario, the test is positive; if at least one validation is expected to fail, the test is negative.

Therefore, in my understanding the book is wrong regarding the "101" example. In such a case, we would have a negative test for an invalid input, which must pass only if the associated validation (in this case, that the typed number is in the 1-100 range) fails.

This also means that, for some input which is not known to be valid or invalid, the behavior is undefined, with no tests for it.

share|improve this answer
1  
You are spekaing about negative/positive in terms of results. But there are categories of tests designed to perform negative operations - trying to break it. E.g. into numeric filed you try to input letters. –  user970696 Mar 29 '13 at 7:16
    
i think in the test case " trying to break it. E.g. into numeric filed you try to input letters" here its a negative test and the app has a bug if you succeed. same principle –  tgkprog Mar 31 '13 at 10:05
    
Yes, filling out a numeric field with letters would be a negative test, since you would have a validation for that. –  Rogério Apr 1 '13 at 11:50
add comment

There is always a lot of semantic confusion when you double your negatives or negate your positives. That's more a limitation of English than of testing theory of methodology. In essence, many scenarios can be described both in what should happen and what shouldn't happen. One is a "positive" effect (what should happen) and one is a "negative" effect - the error. But in actuality these are two different features you are testing.

In your 1-100 example, for instance, the two features are "only accept input between 1 and 100", and "show an error message when input isn't between 1 and 100". These aren't the positive and negative aspects of the same test, but two different tests. The positive element of the first test is to give an input between 1 and 100 and see that the function works. The negative aspect is to give an input of 101 and see that it doesn't work. That's all.

For the second feature, the positive test would be to give an input of 101, or Int32.MinValue, or 0, or other edge cases, and see the error shown. The negative test would be to give an input of 40, and ensure no error is displayed.

share|improve this answer
    
I do not get it..The authors says the field should be between 0-100 and if not, show the error. Then he says showing the error when you type 101 is positive and if the error does not appear, it is negative. Does not make sense IMO. –  user970696 Mar 23 '13 at 17:14
1  
@user970696 In your comment here, "positive" is being used for "the test passes" and "negative" for "the test does not pass". It sounds like that book is not only using nonstandard terminology, but using the same nonstandard words for multiple things. –  Izkata Mar 23 '13 at 17:56
    
@Izkata: exactly, for me as for a non native speaker it is even more confusing..But there is still an issue: what if the requiremenst says e.g. if X is present, do something. Is negative case if X is not present or this is implicit requirement and thus another positive case? –  user970696 Mar 23 '13 at 18:31
    
in VB(A), i frequently use Not x <> "equalval" instead of x = "equalval" because i'm used to sane C-family languages where = always means assignment. they're just different ways of saying the same thing. –  RubyTuesdayDONO Mar 28 '13 at 12:10
1  
Don't you think it's better to get used to the standards and idioms of the language you're using, rather than using an incredibly convoluted and confusing way to do a simple equality check? –  Avner Shahar-Kashtan Mar 28 '13 at 14:00
add comment

You're messing up tests and conditions.

If you look at the illustration in the referred book, you'll see that Koirala and Sheikh do not talk about a decision branch, but a system as a whole.

They only look at valid system tests, i.e., blackbox tests, which results are correct.

So, in this context,

  • a positive test is, when you enter valid data, and the system does process it
  • a negative test is, when you enter invalid data, and the system does not process it (and ideally tells the user about that)

It does not matter, how the validity of the data is determined, or what exactly the wording of the requirement is.

share|improve this answer
    
Right - but if you try to apply that on my examples, it is not that simple. So lets say "When user enters name that equals "James Bond", secret menu appears. Also negative test would be any other name and not opening the menu? –  user970696 Mar 29 '13 at 7:29
1  
No, both are positive tests, since (if) both usernames are valid. They are only processed differently. –  nibra Mar 29 '13 at 16:45
    
well, positive? In this context, the input is invalid because the menu should not appear. As Frank pointed out in his answer - negative (false) are testing conditions not adhering to the specifications. Also in your definiton - "and the system does not process it". That is only in the ideal case but the point of negative test is to provide invalid input and observe. –  user970696 Mar 30 '13 at 14:31
    
What I mean is: Positive test:User enters name James Bond, a menu appears. Negative test: A user enters name James ABC, a menu should not appear. –  user970696 Mar 30 '13 at 14:46
1  
Is "James ABC" invalid from the system's point of view? Or does that user only have less permissions? –  nibra Mar 31 '13 at 1:53
show 4 more comments

Firstly (and probably unintentionally) your examples using * comparisons* tend to confuse the issue., getting the comparison "a <> b" which sounds sort of negativeish confused with the a test of whether the comparison was implemented correctly (a positive test).

So positive tests are where you test to see whether the system implements a use case correctly. (Think of a test for the vending machine use case "customer gets can of coke").

Negative tests are where you test to see if exceptions to a use case are handled properly. (Think of an exception in the above use case "user input foreign coins").

A positive test case passes if the use case succeeds.

A negative test case passes if the use case fails.

and vice versa.

share|improve this answer
add comment

I think the book is trying to make a distinction where it is not strictly necessary.

Tests should be written to make sure a system behaves according to its specification. It should do so for normal operating conditions as well as in exceptional situations.

I assume the author is making a distinction because it is easy to forget to test the exceptional situations. It is just a little guideline that can help you think about things.

Here is an example: If you are writing a function that will multiply a matrix with a scalar, most people will write a test with a demo matrix and some number and check that the multiplication result is correct. The book wants you to also be on the lookout for the case when your matrix might not be initialized or your number might be from the wrong ring.

Knowing or defining if a test case is positive or negative is not really necessary or useful. It's just a guideline to help you avoid missing classes of inputs. It is basically a simple way of creating equivalence classes: http://en.wikipedia.org/wiki/Equivalence_partitioning

share|improve this answer
add comment

"The authors says the field should be between 0-100 and if not, show the error. Then he says showing the error when you type 101 is positive and if the error does not appear, it is negative. Does not make sense IMO."

If a test pass (whatever it test), it is positive (success). If not, its negative (unexpected behavior).

As already said, a test goal is to determine if your code should PASS or FAIL. Failing may be normal. Personally, I categorize my tests with OK (should pass) and KO (should fail) like that :

LoginSuccessTestOK()
{
    // Assert that login success in an expected manner
}

LoginFailTestOK()
{
    // Assert that login fail in an expected manner
}

LoginSuccessTestKO()
{
    // Assert that login should success but fail in an expected manner
}

LoginFailTestKO()
{
    // Assert that login should fail but succeeded in an expected manner
    // May be not applicable in this case
}

If these tests pass (green), they are positive. If one don't (red), it'll be because of an unexpected behavior.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.