Positive and negative is one nomenclature, true and false another. In general, you have to distinguish two things, which are both of a boolean nature:
- Is the test itself written according to the requirements, or violating them? Let me refer to these as true and false from now on.
- Is the test run successful or not? Let me refer to these as positive and negative from now on.
What your quote denotes as "positive" and "negative" are in fact only two of the possible four combinations. The quoted phrase "positive" means a true-positive result, i.e. valid input and a successful test result. Accordingly, "negative" means false-negative, i.e. invalid input and a failed test.
Here's a brief description for each of the four possible combinations and what is hidden behind them:
The simplest case, in which your input is adhering to the specification and the result is as expected. Not much to say about these really.
These are good, too. You have provided valid input, and the failing test tells you that there is something wrong in your product.
These are monsters. They show up in your reports as green checkmarks, but really what the test is testing is not adhering to the specification.
Again you are testing something which doesn't make sense, and you do it in a way that fails. The good point about these is that you at least become aware of the problem by a failing test.
Application to 101 Example
Let's apply this to your 101 example:
First, you write two test cases: Test #1 checks that for valid inputs, the user does not see an error message, whereas Test #2 checks that for input outside of the valid range an error is shown. Both tests adhere to the specification, hence, you can only end up with the true-cases below:
- Test#1 True-positive: I enter 50 and no error shows up
- Test#2 True-positive: I enter 101 and an error shows up
- Test#1 True-negative: I enter 50, but an error shows up
- Test#2 True-negative: I enter 101, but no error shows up
The negative cases both indicate that you have to fix your implementations.
Now consider two more tests: Test #3 checks that when you input a number between 1 and 100, an error message shows up. And Test #4 checks that no error message is shown when you enter 101. Both of these tests are not adhering to the specification! But you can still execute them, because they are just executable tests after all. Here are the possible cases:
- Test #3 False-positive: I enter 50 and an error shows up
- Test #4 False-positive: I enter 101 and no error shows up
- Test #3 False-negative: I enter 50, but no error shows up
- Test #4 False-negative: I enter 101, but an error shows up
As you can see, due to the nature of these tests you will never get a True-X result. The false-negative results are nicer, because they point you to the problem.
Impact on development
Our problem as developers is that we only get to see the result of test executions, for example in an IDE, as success or failure, which maps to the latter positive or negative. You do not get any indication of whether your test is true or false (because it is far from trivial to automatically determine that a test adheres to a specification).
True-negatives are relatively simple to fix: you just fix your code to do what the customer wants it to do.
False-negatives are more trouble: As it is really the test itself, which is at fault, you risk losing valuable time trying to turn a False-negative into a False-positive by "fixing" your product code.
False-positives are not a problem at the time they occur, but further down the road. Imagine that all your tests succeed, but several versions later that one tests suddenly fails. It has been working all the time, so you will assume that some recent code change broke it, but in fact, the test itself was already broken.
You might argue that the False-X variants of tests should not be created - and of course that's right. But they usually do not come into existence, because someone writes them, but rather, because someone wrote them long ago and now someone else has changed the requirements/specification. A simple example would be to change your example to allow 51-150 as valid input. A test that checks for an error for the input 200 remains a True-X test, but the test that tries to enter 101 and expects an error becomes a False-X. Gladly, it will fail.
More problematic is the test that enters 50 and expects no error. Due to the changed specification, 50 is no longer a valid input, but you will not find out that your application still accepts it as valid easily, because your test still says everything is alright. The specification change changed the test from a True-positive into a False-positive.
Lesson to be learned: when changing your specification make sure to update your tests as well.
The nomenclature of true/false-positive/negatives comes from statistics, especially, medical tests. They often have the problem, that tests are not as deterministic in their nature as it is the case for software, so they are also interested in sensitivity and specificity.
Another nomenclature (also in statistics) is to refer to the false-X cases as type I and type II errors.