Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Should the expected results of a unit test be hardcoded, or can they be dependant on initialised variables? Do hardcoded or calculated results increase the risk of introducing errors in the unit test? Are there other factors I haven't considered?

For instance, which of these two is a more reliable format?

[TestMethod]
public void GetPath_Hardcoded()
{
    MyClass target = new MyClass("fields", "that later", "determine", "a folder");
    string expected = "C:\\Output Folder\\fields\\that later\\determine\\a folder";
    string actual = target.GetPath();
    Assert.AreEqual(expected, actual,
        "GetPath should return a full directory path based on its fields.");
}

[TestMethod]
public void GetPath_Softcoded()
{
    MyClass target = new MyClass("fields", "that later", "determine", "a folder");
    string expected = "C:\\Output Folder\\" + string.Join("\\", target.Field1, target.Field2, target.Field3, target.Field4);
    string actual = target.GetPath();
    Assert.AreEqual(expected, actual,
        "GetPath should return a full directory path based on its fields.");
}

EDIT 1: In response to DXM's answer, is option 3 a preferred solution?

[TestMethod]
public void GetPath_Option3()
{
    string field1 = "fields";
    string field2 = "that later";
    string field3 = "determine";
    string field4 = "a folder";
    MyClass target = new MyClass(field1, field2, field3, field4);
    string expected = "C:\\Output Folder\\" + string.Join("\\", field1, field2, field3, field4);
    string actual = target.GetPath();
    Assert.AreEqual(expected, actual,
        "GetPath should return a full directory path based on its fields.");
}
share|improve this question
2  
Do both. Seriously. Tests can and should overlap. Also look into some sort of data-driven tests if you find yourself dealing with hard-coded values. –  Job Dec 23 '11 at 2:19
    
I would agree the third option is what I like to use. I don't think the option 1 would hurt since you eliminate the manipulation on compile. –  kwelch Apr 6 '12 at 15:04

5 Answers 5

up vote 14 down vote accepted

I think calculated expected value results in more robust and flexible test cases. Also by using good variable names in the expression that calculate expected result, it is much more clear where the expected result came from in the first place.

Having said that, in your specific example I would NOT trust "Softcoded" method because it uses your SUT (system under test) as the input for your calculations. If there's a bug in MyClass where fields are not properly stored, your test will actually pass because your expected value calculation will use the wrong string just like target.GetPath().

My suggestion would be to calculate expected value where it makes sense, but make sure that the calculation doesn't depend on any code from the SUT itself.

In Response to OP's update to my response:

Yes, based on my knowledge but somewhat limited experience in doing TDD, I would choose option #3.

share|improve this answer
1  
Good point! Don't rely on the unverified object in the test. –  Hand-E-Food Dec 22 '11 at 23:27
    
isn't it duplication of SUT code? –  Abyx Dec 23 '11 at 12:09
1  
in a way it is, but that's how you verify that SUT is working. If we were to use the same code and it got busted, you would never know. Of course, if in order to perform the calculation, you need to duplicate a lot of SUT, then maybe option #1 would become better, just hardcode the value. –  DXM Dec 23 '11 at 14:26

What if the code was as follows

MyTarget() // constructor
{
   Field1 = Field2 = Field3 = Field4 = "";
}

Your second example wouldn't catch the bug, but the first example would.

In general, I'd recommend against soft-coding because it may hide bugs. For example:

string expected = "C:\\Output Folder" + string.Join("\\", target.Field1, target.Field2, target.Field3, target.Field4);

Can you spot the problem? You wouldn't make that same mistake in a hard coded version. Its harder to get the calculations correct then hard-coded values. That's why I prefer to work with hard coded values then soft-coded ones.

But there are exceptions. What if your code has to run on Windows and Linux? Not only will the path have to be different, it has to use different path separators! Calculating the path using functions that abstract the difference between might make sense in that context.

share|improve this answer
    
I hear what you're saying and that gives me somethign to consider. Softcoding relies on my other test cases (such as ConstructorShouldCorrectlyInitialiseFields) passing. The failure you describe would be cross referenced by other unit tests failing. –  Hand-E-Food Dec 22 '11 at 23:00
    
@Hand-E-Food, it sounds like you are writing tests on individual methods of your objects. Don't. You should write tests that check the correctness of your whole object together not individual methods. Otherwise your tests will be brittle with respect to changes inside the object. –  Winston Ewert Dec 22 '11 at 23:13
    
I'm not sure I follow. The example I gave was purely hypothetical, an easy to understand scenario. I'm writing unit tests to test public members of classes and objects. Is that the correct way to use them? –  Hand-E-Food Dec 22 '11 at 23:31
    
@Hand-E-Food, if I understand you correctly, your test ConstructShouldCorrectlyInitialiseFields would invoke the constructor and then assert that the fields are set correctly. But you shouldn't do that. You shouldn't care what the internal fields are doing. You should only assert that the external behavior of the object is correct. Otherwise, the day may come when you need to replace the internal implementation. If you've made assertions about the internal state, all your tests will break. But if you've only made assertions about external behavior, everything will still work. –  Winston Ewert Dec 23 '11 at 1:08
    
@Winston--I'm actually in the process of plowing through xUnit Test Patterns book and before that finished The Art of Unit Testing. I'm not going to pretend I know what I'm talking about, but I'd like to think I picked up something from those books. Both books strongly recommend that each test method should test absolute minimum and you should have many test cases to test your whole object. That way when interfaces or functionality changes, you should only expect to fix few test methods, rather than most of them. And since they are small, changes should be easier. –  DXM Dec 23 '11 at 1:55

In my opinion, both of your suggestions are less than ideal. The ideal way to do it is this one:

[TestMethod]
public void GetPath_Hardcoded()
{
    const string f1 = "fields"; const string f2 = "that later"; 
    const string f3 = "determine; const string f4 = "a folder";

    MyClass target = new MyClass( f1, f2, f3, f4 );
    string expected = "C:\\Output Folder\\" + string.Join("\\", f1, f2, f3, f4);
    string actual = target.GetPath();
    Assert.AreEqual(expected, actual,
        "GetPath should return a full directory path based on its fields.");
}

In other words, the test should work exclusively based on the input and the output of the object, and not based on the internal state of the object. The object should be treated as a black box. (I disregard other issues, like the inappropriateness of using string.Join instead of Path.Combine, because this is just an example.)

share|improve this answer
1  
Not all methods are functional--many correctly have side effects that change the state of some object or objects. A unit test for a method with side effects would probably need to evaluate the state of the object(s) affected by the method. –  Matthew Flynn Dec 22 '11 at 23:18
    
Then that state would be considered as the output of the method. The intent of this sample test is to check the GetPath() method, not the constructor of MyClass. Read @DXM's answer, he provides a very good reason for taking the black box approach. –  Mike Nakis Dec 22 '11 at 23:25
    
@MatthewFlynn, then you should the test the methods affected by that state. The exact internal state is an implementation detail and none of the test's business. –  Winston Ewert Dec 22 '11 at 23:28
    
@MatthewFlynn, just to clarify, is that related to the example shown, or something else to consider for other unit tests? I could see that mattering for something like target.Dispose(); Assert.IsTrue(target.IsDisposed); (a very simple example.) –  Hand-E-Food Dec 22 '11 at 23:35
    
Even in this case, the IsDisposed property is (or should be) an indispensable part of the public interface of the class, and not an implementation detail. (The IDispose interface does not provide such a property, but that's unfortunate.) –  Mike Nakis Dec 22 '11 at 23:38

There are two aspects in the discussion:

1. Using the target itself for test case
The first question is should/can you use the class itself to rely and get part of the work done in the test stub? - The answer is NO since, in general, you should never make assumption about the code which you are testing. If this is not done properly, over time bugs become immune to some unit testing.

2. Hardcoding
should you hard code? Again the answer is No. because like any software - the hard coding of he information becomes difficult when things evolve. For example, when you want the above path to be modified again, you need to either write additional unit or keep modifying. A better method is to keep input and evaluation date derived from the separate configuration that can be easily adapted.

for example here is how i would right the test stub.

[TestMethod]
public void GetPath_Tested(int CaseId)
{
    testParams = GetTestConfig(caseID,"testConfig.txt"); // some wrapper that does read line and chops the field. 
    MyClass target = new MyClass(testParams.field1, testParams.field2);
    string expected = testParams.field5;
    string actual = target.GetPath();
    Assert.AreEqual(expected, actual,
        "GetPath should return a full directory path based on its fields.");
}
share|improve this answer

There are a lot of concepts possible, made some examples to see the difference

[TestMethod]
public void GetPath_Softcoded()
{
    //Hardcoded since you want to see what you expect is most simple and clear
    string expected = "C:\\Output Folder\\fields\\that later\\determine\\a folder";

    //If this test should also use a mocked filesystem it might be that you want to use
    //some base directory, which you could set in the setUp of your test class
    //that is usefull if you you need to run the same test on different environments
    string expected = this.outputPath + "fields\\that later\\determine\\a folder";


    //another readable way could be interesting if you have difficult variables needed to test
    string fields = "fields";
    string thatLater = "that later";
    string determine = "determine";
    string aFolder = "a folder";
    string expected = this.outputPath + fields + "\\" + thatLater + "\\" + determine + "\\" + aFolder;
    MyClass target = new MyClass(fields, thatLater, determine, aFolder);

    //in general testing with real words is not needed, so code could be shorter on that
    //for testing difficult folder names you write a separate test anyway
    string f1 = "f1";
    string f2 = "f2";
    string f3 = "f3";
    string f4 = "f4";
    string expected = this.outputPath + f1 + "\\" + f2 + "\\" + f3 + "\\" + f4;
    MyClass target = new MyClass(f1, f2, f3, f4);

    //so here we start to see a structure, it looks more like an array of fields
    //so what would make testing more interesting with lots of variables is the use of a data provider
    //the data provider will re-use your test with many different kinds of inputs. That will reduce the amount of duplication of code for testing
    //http://msdn.microsoft.com/en-us/library/ms182527.aspx


    The part where you compare already seems correct
    MyClass target = new MyClass(fields, thatLater, determine, aFolder);

    string actual = target.GetPath();
    Assert.AreEqual(expected, actual,
        "GetPath should return a full directory path based on its fields.");
}

To summarise: In general your first just hardcoded test makes most sense to me because it is simple, straight to the point etc. If you start hardcoding a path too many times just put it in the setup method.

For more future structured testing I would go to check out datasources so you can just add more data rows if you need more testing situations.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.