Tag Info

Hot answers tagged

128

If every path through a program is tested, does that guarantee finding all bugs? No If not, why not? How could you go through every possible combination of program flow and not find the problem if one exists? Because even if you test all possible paths, you still haven't tested them with all possible values or all possible combinations of values. ...


63

In addition to Mason's answer, there is also another problem: coverage does not tell you what code was tested, it tells you what code was executed. Imagine you have a testsuite with 100% path coverage. Now remove all assertions and run the testsuite again. Voilà, the testsuite still has 100% path coverage, but it tests absolutely nothing.


32

Here's a simpler example to round things off. Consider the following sorting algorithm (in Java): int[] sort(int[] x) { return new int[] { x[0] }; } Now, let's test: sort(new int[] { 0xCAFEBABE }); Now, consider that (A) this particular call to sort returns the correct result, (B) all code paths have been covered by this test. But, obviously, the ...


11

Consider the abs function, that returns the absolute value of a number. Here is a test (Python, imagine some test framework): def test_abs_of_neg_number_returns_positive(): assert abs(-3) == 3 This implementation is correct, but it only gets 60% code coverage: def abs(x): if x < 0: return -x else: return x This ...


7

Yet another addition to Mason's answer, a program's behavior may depend on the runtime environment. The following code contains a Use-After-Free: int main(void) { int* a = malloc(sizeof(a)); int* b = a; *a = 0; free(a); *b = 12; /* UAF */ return 0; } This code is Undefined Behavior, depending on the configuration ...


6

It's clear from the other answers that 100% code coverage in tests does not mean 100% code correctness, or even that all bugs that could be found by testing, will be found (never mind bugs that no test could catch). Another way of answering this question is one from practice: There are, in the real world, and indeed on your own computer, many pieces of ...


4

Path coverage cannot tell you whether all the required features have been implemented. Leaving out a feature is a bug, but path coverage will not detect it.


4

Part of the issue is that 100% coverage only guarantees that the code will function correctly after a single execution. Some bugs like memory leaks may not be apparent or cause issue after a single execution, but over time will cause problems for the application. For example, say you have an application which connects to a database. Perhaps in one method ...


4

These two types of testing are not directly related. A smoke test is a quick system test with the purpose of finding major flaws in a software artifact. For example, a test might deploy a web app to a test server, validate that it deploys and starts up, and the server can service simple requests. This would validate that the application file (e.g. WAR file ...


4

Basically, in smoke testing you turn on the system, especially for the first time ever, and see whether it starts to emit smoke... which would be a sign that something is very wrong with it. It doesn't tell you anything beyond that, but it can still be valuable to avoid further effort on something that is obviously not going to work at all. According to ...


3

If every path through a program is tested, does that guarantee finding all bugs? As already said, the answer is NO. If not, why not? Besides what is being said, there are bugs appearing at different levels, which can't be tested with unit tests. Just to mention few : bugs caught with integration tests (unit tests shouldn't use real resources ...


2

First, I think this is a perfectly good question. It's an interesting area and i don't get why was this downvoted. You don't need winsock you can disable the network a adapter and change it's settings programmatically... but network issues could be very tricky to simulate this way... ...


2

Multiple VMs is the way to go, but you should not really have to "login to each VM to run the test suite": you should be able to automate that part. You can set up all VMs to look at a common shared folder of the host machine, and run a process on each VM which checks the contents of that folder, say, once a second. When a new version of the app is ...


2

In general, your tests should normally not reimplement the code under test. This can happen to you with traditional tests as well. In fact, from my experience, property-based testing tends to be better for avoiding re-implementations. When you think about the properties, you specifically do not think about implementation details (i.e. do not think about how ...


2

Yes selenium can be used to automate the manual interaction steps. It's intended to be used against web browsers and it's widely used and implemented. However this is only half of the puzzle. Selenium is the core technology in browser automation tools, APIs and frameworks. To actually USE selenium you are going to need to pick an actual implementation. ...


2

What does it mean for every path to be tested? The other answers are great, but I just want to add that the condition "every path through a program is tested" is itself vague. Consider this method: def add(num1, num2) foo = "bar" # useless statement $global += 1 # side effect num1 + num2 # actual work end If you write a test that asserts add(1, ...


1

What strategy would one adopt to test such a form? You have to separate the GUI from the decision logic by adding a decision function that only depends on business attributes and not on GUI elements: bool isNativeLanguageInputVisible(RegistrationData currentRegistrationData) Your GUI would use this decision function to control visibility and your ...


1

I don't know about "in terms of reliability testing", but the meanings of those to terms is as follows: Calendar time - How long whatever you are testing took in actual time. In other words, if you start a stopwatch when you start a program and check it when the program is done, you are measuring calendar time. This is equivalent to the "real" time ...


1

This is the driving motivation behind Test-driven development, with the principles being thus: You do not write code without tests. You write just enough code for the test to pass. You refactor the code that you have written and ensure that tests pass. I think I understand what you're getting at - you don't know if you can/should write tests for objects ...


1

Beyond mocking the network interface, you don't need two physical machines for a system test: Disable/reenable the network card using scripts that do ifconfig/ipconfig. use a virtual machine for one end of the system and script it up/down or enable/disable the networking. Hooking into WinSock is way more work than you need to do here.


1

You test these by mocking the client and server and using IP Address 127.0.0.1 Depending on the test you are performing, it could be a simple matter of having the mock-server simply closing its end of the connection after some predefined time and then verifying that the client reconnected within the specified time-frame. It may be difficult to test some of ...


1

Black-box testing means that the tester is not familiar with the inner workings of the system, while white-box testing is performed by someone who knows exactly how the program works. Both has advantages and disadvantages. Functional testing is testing whether or not the software product fulfills functional requirements, like "when loading an incorrectly ...


1

Black-box testing as opposed to white-box testing is a broad category of testing that includes tests where you do not know, and do not need to know, how the implementation works, you just want to test the outcome. For example: I click the buy button on a web site and the item I selected is now in my shopping cart. I don't care how the programmer made this ...


1

One thing nobody else has mentioned yet: mutation testing. This is where an automated tool takes your source code and deliberately inserts bugs into it. (E.g., delete a randomly-chosen statement, change an AND to an OR, or whatever.) It then runs your full test suite, and checks whether the tests pass. If all tests pass, then there are two possibilities: ...


1

This is the kind of thing you definitely need to integration test, as real-world file systems have all kind of strange behavior (like the way Windows won't allow deleting a file if any process, including the deleter, has it open). So the TDD approach is to write the integration test first (TDD, strictly speaking, doesn't have distinct concepts of 'unit ...



Only top voted, non community-wiki answers of a minimum length are eligible