Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I work in a small company as a solo developer. I'm the only developer at the company in fact. I have several (relatively) large projects I've written and maintain regularly, and none of them have tests to support them. As I begin new projects I often wonder if I should try a TDD approach. It sounds like a good idea, but I honestly can never justify the extra work involved.

I work hard to be forward-thinking in my design. I realize that certainly one day another developer will have to maintain my code, or at least troubleshoot it. I keep things as simple as possible and I comment and document things that would be difficult to grasp. And the fact is these projects aren't so big or complicated that a decent developer would struggle to comprehend them.

A lot of the examples I've seen of tests get down to the minutiae, covering all facets of the code. Since I'm the only developer and I'm very close to the code in the entire project, it is much more efficient to follow a write-then-manually-test pattern. I also find requirements and features change frequently enough that maintaining tests would add a considerable amount of drag on a project. Time that could otherwise be spent solving the business needs.

So I end up with the same conclusion each time. The return on investment is too low.

I have occasionally setup a few tests to ensure I've written an algorithm correctly, like calculating the number of years someone has been at the company based on their hire date. But from a code-coverage standpoint I've covered about 1% of my code.

In my situation, would you still find a way to make unit testing a regular practice, or am I justified in avoiding that overhead?

UPDATE: A few things about my situation that I left out: My projects are all web applications. To cover all my code, I'd have to use automated UI tests, and that is an area where I still don't see a great benefit over manual testing.

share|improve this question
1  
Thanks everyone. I'm learning a lot here. A few things about my situation that I left out: My projects are all web applications. To cover all my code, I'd have to use automated UI tests, and that is an area where I still don't see a great benefit over manual testing. –  Ken Pespisa Apr 8 '11 at 16:54
    
Automated UI tests are difficult. Especially because of all the browser vagaries and quirks. Also, don't add information in comments. Update your question with additional facts. –  S.Lott Apr 8 '11 at 17:41
1  
We're having great success at Transactis using Telerik's web automation testing tool. We've already got dozens of formerly manual browser tests converted to automation. The automated tests are WAY faster, and are also GREAT for highlighting any performance issues your web site may have. –  John Kaster Apr 9 '11 at 5:51
    
1  
I have seen a project which tried to have automated browser testing of complete web pages. As far as i can tell, it has not found any of the hundreds of severe bugs we found through manual testing, and it cost an enormous amount of time to develop and maintain. (Using Selenium driven by NUnit). Worse, some of the tests break frequently for non-problems, due to browser and test framework incompatibilities. –  O'Rooney Nov 16 '13 at 18:50

13 Answers 13

up vote 46 down vote accepted

A lot of the examples I've seen of tests get down to the minutiae, covering all facets of the code.

So? You don't have to test everything. Just the relevant things.

Since I'm the only developer and I'm very close to the code in the entire project, it is much more efficient to follow a write-then-manually-test pattern.

That's actually false. It's not more efficient. It's really just a habit.

What other solo developers do is write a sketch or outline, write the test cases and then fill in the outline with final code.

That's very, very efficient.

I also find requirements and features change frequently enough that maintaining tests would add a considerable amount of drag on a project.

That's false, also. The tests are not the drag. The requirements changes are the drag.

You have to fix the tests to reflect the requirements. Whether their minutiae, or high-level; written first or written last.

The code's not done until the tests pass. That's the one universal truth of software.

You can have a limited "here it is" acceptance test.

Or you can have some unit tests.

Or you can have both.

But no matter what you do, there's always a test to demonstrate that the software works.

I'd suggest that a little bit of formality and nice unit test tool suite makes that test a lot more useful.

share|improve this answer
3  
I like your first statement, to test just the relevant things. Regarding the efficiency of manual vs unit testing, I don't believe my statement is entirely false, nor your's is entirely true. Seems there's a balance to be found between automatic and manual testing to achieve maximum efficiency. –  Ken Pespisa Apr 8 '11 at 16:38
4  
@Ken Pespisa: Sorry. I drank the TDD Kool-Aid about two years ago (after 30 years of test-last). Now I'm stuck on test-first. It made me much, much more productive because I have less thinking to do when building. –  S.Lott Apr 8 '11 at 16:42
2  
@Ken Pespisa: No. There's balance in the answers. If you asked if it's right to divide by zero, you'd get overwhelming responses leaning one way for a reason. If you asked if sqrt(-1) should be an integer, you'd get overwhelming responses leaning one way. The balance is around "how" and "what order". The bare fact is that you must test. So write the tests first and make sure they work. –  S.Lott Apr 8 '11 at 17:02
13  
Consider unit testing the interface not the implementation details. Test the limits and border cases. Test the risky code. Lots of code is simple enough to verify by inspection, although inspecting your code is more error prone than inspecting someone else's code. The first time manual tests may be more efficient, by the tenth time automated tests are way ahead. –  BillThor Apr 8 '11 at 17:21
3  
"The code's not done until the tests pass" - Not really, IMO. Code starts when it passes the tests. Code is not done until it's been live for a year or two and been subject to stress tests and integration tests with large, active and impatient user base. That is the only testing that really counts. –  Vector Jun 16 '13 at 5:15

You have to balance the cost of testing with the cost of bugs.

Writing a 10 line unit test for a function that opens a file, where the failure is "file not found" is pointless.

A function that does something complex to a complex data structure - then obviously yes.

The tricky bit is in-between. But remember the real value of unit tests isn't testing the particular function, it's testing the tricky interactions between them. So a unit test that spots that a change in one bit of code, breaks some function in a different module a 1000 lines away, is worth its weight in coffee.

share|improve this answer
1  
Excellent - thank you. –  Ken Pespisa Apr 8 '11 at 17:03

Testing is gambling.

Creating a test is a bet that the cost of bugs in a unit occurring and not catching them (now, or during all future code revisions) is greater than the cost of developing the test. Like any bet, sometimes you win, sometimes you lose.

Sometime late software with much less bugs wins out over quick but buggy stuff that gets to market first. Sometimes the opposite. You have to look at the statistics in your particular field, and how much management wants to gamble.

Some types of bugs might be so unlikely to be made, or to make it out of any early sanity testing, as to be statistically not worth the time to create additional specific tests. But sometimes the cost of a bug is so great (medical, nuclear, etc.) that a company must take a losing bet, similar to buying insurance. Many apps do not have such a high failure cost, and don't need higher insurance coverage.

share|improve this answer
2  
Good answer. One of the few that really answers my original question. I have submerged myself into the world of testing since writing this post (I like it btw.) I need to understand it more before I can truly know when to use it (or not). For many of the reasons stated here I would prefer to use it all the time. But it will ultimately depend on how much faster I get at it, because in the end it is a gamble of my time, which is under my company / client's control, and often they focus on the bottom corners of the project triangle: en.wikipedia.org/wiki/Project_triangle –  Ken Pespisa Apr 27 '11 at 1:09

My advice is to only test the code that you want to work properly.

Don't test the code that you want to be buggy and to cause problems for you down the road.

share|improve this answer
3  
Reminds me of the saying my dentist has: You don't have to floss all of your teeth, just the ones you want to keep. –  Ken Pespisa Apr 9 '11 at 23:55
    
I confess that is what made me think of it. ;-) –  Nick Hodges Apr 10 '11 at 23:45

Imagine that you had a suite of tests that could run in an eyeblink and would light up a green or red light. Imagine that this suite of tests tested everything! Imagine that all you had to do to run the suite of tests was to type ^T. What power would this give you?

Could you make a change to the code without fear of breaking something? Could you add a new feature without fear of breaking an old feature? Could you clean up messy code quickly without fear of doing damage?

Yes, you could do all those things! And what would happen to your code over time? It would get cleaner and cleaner because there would be no risk to cleaning it.

Let's imagine that you had a little fairy on your shoulder. Every time you wrote a line of code, the fairy would add something to the test suite that tested that that line of code did what it was intended to do. So every couple of seconds you could hit ^T and see that the last line of code you wrote worked.

How much debugging do you think you would do?

If this sounds like fantasy, you're right. But the reality is not much different. Replace the eyeblink with a few seconds, and the fairy with the TDD discipline, and you've pretty much got it.

Let's say you are coming back to a system you built a year ago, and you've forgotten how to create one of the central objects. There are tests that create that object every way it can be created. You can read those tests and jog your memory. Need to call an API? There are tests that call that API every way it can be called. These tests are little documents, written in a language you understand. They are completely unambiguous. They are so formal that they execute. And they cannot get out of sync with the application!

Not worth the investment? You've got to be kidding me! How could anyone NOT want that suite of tests? Do yourself a favor and stop quibbling over silliness. Learn to do TDD well, and watch how much faster you go, and how much cleaner your code is.

share|improve this answer
9  
Wow, THE Uncle Bob? It's great to get your thoughts here. I agree with you on the benefits of TDD, there really is no argument to be had there. The question is about the investment of time and the ROI. It isn't silly for me to consider these things. Imagine a project will take me 50% more time to finish with TDD than without, and the fairy tells me it will only save me 10% time over manual testing in the lifetime of the project. That might seem like a fantasy, but I see it as entirely plausible with certain projects. –  Ken Pespisa Apr 9 '11 at 1:34
6  
@Ken "Imagine a project will take me 50% more time to finish with TDD than without". That sounds EXACTLY like fantasy to me. In fact, it sounds like you just made that figure up on the spot without a shred of evidence to support it. –  Rein Henrichs Apr 29 '11 at 5:11
9  
@Rein Henrichs - Of course I made the number up, it was a hypothetical statement. I'm making the point that TDD adds a significant amount of time to a project, and I have to consider whether I'm going to get something of equal or better value in return. You don't have to convince me on the values of TDD, I'm convinced. But it is not a panacea. –  Ken Pespisa Apr 29 '11 at 16:09
6  
@Rein, what exactly is the "available evidence?" Please elaborate. –  Ken Pespisa May 23 '11 at 22:15
8  
@Uncle Bob "Replace the eyeblink with a few seconds": You are joking, of course. TDD is a good tool, but you have to test only the relevant parts otherwise you spend more time maintaining tests than doing any serious development. This is especially true when requirements are changing very quickly: you are constantly writing and throwing away tests for classes that change all the time. I am not saying TDD is bad, it must just be used sensibly and not applied mechanically as you seem to suggest. –  Giorgio Oct 10 '13 at 20:37

I worked on a system where we tested almost everything. The notable executions to testing were the PDF and XLS output code.

Why? We were able to test the parts that gathered the data and built the model that was used to create the output. We were also able to test the parts that figured out what parts of the model would go to the PDF files. We weren't able to test if the PDF looked ok because that was totally subjective. We weren't able to test that all the parts in a PDF were readable to a typical user because that was also subjective. Or if the choice between bar and pie charts was correct for the dataset.

If the output is going to be subjective, there is little unit testing you can do what is worth the effort.

share|improve this answer
    
Actually, this kind of testing is probably "integration testing". And yes, integration testing is a lot harder than unit testing, and one reason is that sometimes the rules for what is "correct" are very complicated, or even subjective. –  sleske Sep 6 at 22:01

Ken, I and many many other developers out there have arrived at the same conclusion as you several times throughout our careers.

The truth that I believe you'll find (as have many others) is that the initial investment of writing tests for your application may seem daunting, but if written well and targetted at the correct parts of your code, they can really save a ton of time.

My big problem was with the testing frameworks available. I never really felt like they were what I was looking for, so I just rolled my own very simple solution. It really helped to bring me around to the "dark side" of regression testing. I'll share a basic pseudo snippet of what I did here, and hopefully you can find a solution that works for you.

public interface ITest {
    public string Name {
        get;
    }
    public string Description {
        get;
    }
    public List<ITest> SubTests {
        get;
    }
    public TestResult Execute();
}

public class TestResult {
    public bool Succesful {
        get;
        set;
    }

    public string ResultMessage {
        get;
        set;
    }

    private Dictionary<ITest, TestResult> subTestResults = new Dictionary<ITest, TestResult>();
    public Dictionary<ITest, TestResult> SubTestResults {
        get {
            return subTestResults;
        }
        set {
            subTestResults = value;
        }
    }
}

The only tricky part after that is figuring out what granularity level you think is the best "bang for the buck" for whatever project you're doing.

Building an address book will obviuosly require way less testing than an enterprise search engine, but the fundamentals don't really change.

Good luck!

share|improve this answer
    
I think with time I'll figure out what granularity level is best. It's good to hear from others that are regularly creating tests that they approach it sensibly and not robotically write tests for every conceivable outcome. My first introduction to tests was on that kind of level where everything-must-be-tested. In fact, the whole concept of TDD seems to follow that mantra. –  Ken Pespisa Apr 8 '11 at 17:08
    
I think you might benefit from using a Framework like SubSpec (it's BDD inspired), so that will allow you to get Assert ("SubTest") Isolation whilst sharing context setup. –  Johannes Rudolph Apr 13 '11 at 12:59

Professional developers write unit tests because, in the longer term, they save time. You are going to test your code sooner or later, and if you don't your users will, and if you have to fix bugs later they are going to be harder to fix and have more knock on effects.

If you are writing code with no tests and have no bugs then fine. I don't believe you can write a non-trivial system with zero bugs though, so I assume you are testing it one way or another.

Unit tests are also crucial to prevent regressions when you modify or refactor older code. They do not prove your change hasn't broken old code but they give you a lot of confidence (so long as they pass of course:) )

I would not go back and write a whole batch of tests for code that you have already shipped, but next time you need to modify a feature I'd suggest trying to write tests for that module or class, get your coverage up to 70%+ before you apply any changes. See if it helps you.

If you try it and can honestly say it was no help then fair enough, but I think there is enough industry evidence that they help to make it at least worth your while trialling the approach.

share|improve this answer
    
I like the thoughts on preventing regressions and adding confidence. Those are exactly the reasons I would want to add tests. –  Ken Pespisa Apr 8 '11 at 16:44

The guys at JUnit (Java Unit test framework) have a philosophy that if it is too simple to test, don't test it. I highly recommend reading their Best Practices FAQ, as it is fairly pragmatic.

TDD is a different process of writing your software. The basic premise behind unit testing is that you will spend less time in the debugger stepping through code, and more quickly figure out if your code change accidentally breaks something else in the system. That fits in with TDD. The TDD cycle is like this:

  1. Write a test
  2. Watch it fail (prove you have something to do)
  3. Write just what is needed to make the test pass--no more.
  4. Watch it pass (yay!)
  5. Refactor (make it better)
  6. Wash, rinse, and repeat

What is less obvious about applying TDD is that it changes the way your write code. By forcing yourself to think about how to test/validate that the code is working, you are writing testable code. And since we are talking unit testing, that usually means that your code becomes more modular. To me, modular and testable code is a big win up front.

Now, do you need to test things like C# properties? Imagine a property defined like this:

bool IsWorthTesting {get; set;}

The answer would be "no" it's not worth testing, because at this point you are testing the language feature. Just trust that the C# platform guys got it right. Besides, if it failed, what could you do to fix it?

Also, you will find that there are certain parts of your code that very well will be too much effort to test properly. That means don't do it, but make sure you test the code that uses/is used by the tricky problem:

  • Checked exceptions that can only occur if an install went bad. Java has a ton of these. You are required to write a catch block or declare the checked exception even if there is no way it can fail without hacking the installed files.
  • User interfaces. Finding the control under test, and invoking the right events to simulate a user's actions are very troublesome, and in some cases impossible. However, if you use the Model/View/Controller pattern, you can make sure your model and controllers are tested and leave the view part to manual testing.
  • Client/server interactions. This is no longer a unit test, and is now an integration test. Write all the parts that go up to sending and receiving messages over the wire, but don't actually go over the wire. A good approach is to reduce the responsibility of the code that actually talks over the wire to the raw communications. In your unit test code, mock communication object out to make sure the services are behaving as you expect.

Believe it or not, TDD will help you fall into a sustainable pace of development. It's not because of magic, but rather because you have a tight feedback loop and you are able to catch really dumb mistakes quickly. The cost of fixing those mistakes is essentially constant (at least enough for planning purposes) because the small mistakes never grow up to be big mistakes. Compare that with the bursty nature of code binge/debug purge sprints.

share|improve this answer
1  
i think this answer deserve to be the accepted answer :) –  Karim May 23 '12 at 15:18

I often wonder if I should try a TDD approach. It sounds like a good idea, but I honestly can never justify the extra work involved.

TDD and Unit Testing are not the same thing.

You can write code, then add unit tests later. That is not TDD, and is a lot of extra work.

TDD is the practice of coding in a loop of Red Light. Green Light. Refactor iterations.

This means writing tests for code that does not yet exist, watching the tests fail, fixing the code to make the tests work, then making the code "right". This often saves you work

One of the advantages to TDD is that it reduces the need to think about trivia. Things like off-by-one errors disappear. You don't have to go hunting through API documentation to find out if the list it returns starts at 0 or 1, just do it.

share|improve this answer
    
Could you elaborate on how off-by-one errors disappear? Are you saying that you can more quickly get your answer on whether an array's index is zero-based or one-based via testing than searching through documentation? Seems unlikely to me - I'm pretty fast on Google :) –  Ken Pespisa Apr 8 '11 at 16:41
    
Actually, writing TDD's an excellent way to explore an API (including a legacy codebase, for the purposes of documenting functionality). –  Frank Shearar Apr 8 '11 at 17:00
    
It's also very useful if that API ever changes... You suddenly have some failing tests :-) –  Levinaris Apr 8 '11 at 19:05
    
@Ken Pespisa, It's definitely quicker - write the code based on whether you think it's 0 or 1, run it, fix it if needed. Most of the time, you'll be right and you'd have skipped having to look it up, if you are wrong, you know within 10 seconds. –  Paul Butcher Apr 9 '11 at 15:55
    
Very interesting benefit. I kind of like that. –  Ken Pespisa Apr 9 '11 at 23:53

The mistake you are making is that you are seeing testing as a time investment with no immediate return. It doesn't necessarily work like that.

Firstly writing tests really focusses you on what this part of your code needs to do.

Secondly running them shows up bugs that would otherwise come up in testing.

Thirdly running them sometimes shows up bugs that wouldn't otherwise come up in testing and then would really bite you in the ass in production.

Fourthly if you hit a bug with a system that is running and create a unit test for it, you will not be able to re-introduce that bug later. That can be a really big help. Reintroduced bugs are common and very annoying.

Fifthly if you ever need to hand code over to someone else, a test suite will make their life far easier. Also if you have ignored a project and come back to it after a few years, you won't be so close to it any more and it will be helpful to you as well.

My experience has consistently been that across the development of a project, having decent unit tests has always made the process quicker and more reliable.

share|improve this answer
    
+1 For putting the "Firstly" clause first and foremost. It's not called test-driven for nothing. –  pdr Apr 8 '11 at 16:10
    
I really like your fourth and fifth points. –  Ken Pespisa Apr 8 '11 at 16:17
    
@Ken, test suites are specifications in an executable form. –  user1249 Apr 26 '11 at 20:29

Two very good answers i have come across are in these

  1. When to unit-test vs manual test
  2. What not to test when it comes to Unit Testing?

Justification for avoiding percieved overhead?

  • Time/Cost Immediate saving for your company
  • Potential Time/cost saving in troubleshooting /maintainability/extension the long run even after your gone.

Would you not want to leave a great product from your side as the proof of your quality work? Speaking in selfish terms ,is it not better for you that you do?

share|improve this answer
1  
Good question at the end. I take pride in my work, absolutely, and my applications work very well (if I may be so bold). But you're right - they could be even better with the support of some tests. I think my take away here is to try to fit in as many useful tests as possible within the time I have to work on the project, and without getting too obsessed about code coverage. –  Ken Pespisa Apr 8 '11 at 17:01

For many things, a 'write-then-manually-test' takes no more time than writing a couple tests. The time savings come from being able to re-run those tests at any time.

Think of it: If you have some decent feature coverage with your tests (not to be confused with code coverage), and let's say you have 10 features - clicking a button means you have roughly, 10 yous re-doing your tests... while you sit back and sip your coffee.

You also don't have to test the minutae. You can write integration tests that cover your features if you don't want to get down to the nitty gritty details... IMO, some unit tests get too fine-grained testing the language and platform, and not the code.

TL;DR It's really never appropriate because the benefits are just too good.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.