18

I had planned to study and apply unit testing to my code, but after talking with my colleagues, some of them suggested to me that it's not necessary and it has a very little benefit. They also claim that only a few companies actually do unit testing with production software.

I am curious how people have applied unit testing at work and what benefits they are getting from using them, e.g., better code quality, reduced development time in the long term, etc.

10
  • 13
    "They also claim that only few companies do unit-test with production software." That means that only a few companies produce high-quality software. Which group do you want to align yourself with? How can "only a few companies" be a negative statement? Only a few companies are wildly profitable; does that make it bad? Only a few companies are truly delightful places to work; does that make it bad? Only a few companies minimize their creation of waste; does that make it bad? Their "only a few companies" comment is senseless.
    – S.Lott
    Jun 30, 2011 at 9:53
  • 2
    its also probably incorrect, anecdotally, I've yet to work for or with a company that didn't at least do some level of unit testing
    – jk.
    Jun 30, 2011 at 11:11
  • 1
    I would posit that a programmer that thinks unit testing "has a very little benefit" is inexperienced, or simply doesn't know how to write effective unit tests.
    – Toby
    Jun 30, 2011 at 12:45
  • How much unit testing do the developers of this site use?
    – JeffO
    Jun 30, 2011 at 14:09
  • 1
    @S.Lott : senseless yes, yet, it is still wielded as an argument not to do unit test by those who do not want to do them. I'm not defending their point of view here, quite the contrary. However that statement is still made and those who make it are convinced of it's value and since it is them we must convince of the actual benefits of testing brushing it off as senseless is not going to help much the greater cause.
    – Newtopian
    Jun 30, 2011 at 17:16

14 Answers 14

20

some of them suggest me that it's not necessary

Well, in the strict sense they are right: testing, code reviews, source control etc. are not strictly necessary either. Only that very few sensible developers do work without these in the long term. Most of us have learnt (often the hard way, from our own mistakes) why these are best practices.

and it has a very little benefit.

This is not true in most cases. That is, if we are talking about production software which is supposed to be in use - thus in maintenance - for years to come.

They also claim that only few companies do unit-test with production software.

This may be true (albeit in my experience less and less), but it has nothing to say about the value of unit testing. Contrast to this the old joke "Shit is good - 50 billion flies can't be wrong!" ;-)

did you apply unit-test at your work and what do you get from it? (e.g. better code quality, reduce time in long term etc.)

I have been applying unit tests in my work, whenever it was possible, for over 10 years now. That feeling of confidence in my code, knowing that it works - and I can prove it anytime, anywhere, in a few seconds, again and again, as opposed to just believing in it - is priceless. It gives me the courage to refactor my code, to improve its design, or to fix bugs, without the fear of breaking existing functionality. I would never go back to those old days of "code and pray".

14

I do unittests, but not (or few) at work

The main benefits of testing i get are, that refactoring is a whole lot easier and that regression (i.e., reintroduction of errors already fixed) is noticed.

Testing lives with 'the build': you check out software with running tests, and you dont check in until all the tests are running with your modification.

So, in my experience, writing tests is nearly useless when done in a lone-wolf fashion. When you always have to fix your tests to accomodate for the changes the others made, when the tests may get deleted by your coworkers (happend in my previous jobs) because 'they wont let me deploy', etc. they are only extra work with the benefits instantly burned by the rest of the team.

When the whole team agrees (and, in a team of 1 that is easy), in my experience, tests are a big benefit for quality, style and robustness of the Project.

But in a team that honestly believes that 'only a few companies test their code automatically' and are convinced that they are a waste of time anyway, there seems to be little hope to earn the benefits, but a big possibility that you will be made responsible for the 'loss of time' when pointing out errors.

The best chance to introduce tests would be a small project within your sole responsibility. There you would have the opportunity to shine and to demonstrate the value of tests.

9
  • 1
    Sorry for sounding negative, but i am still burned by 'how to get things done as a grunt' ;)
    – keppla
    Jun 30, 2011 at 8:25
  • Very good advice. The problem is that when done right, you don't actually see the benefit of unit testing. You only potentially see the harm when not doing unit testing. So if you need to convince others with statistics, rather than theory, you're pretty much screwed. Jun 30, 2011 at 8:36
  • 12
    @keppla, I have been writing unit tests in projects where my teammates haven't even heard of unit testing before. Once my tests managed to catch a few bugs which would otherwise have slipped through unnoticed, the others started to take notice and soon they wanted to learn unit testing themselves. Concrete evidence is king. Jun 30, 2011 at 8:45
  • 1
    @keppla, fair enough, if teammates are biased to the level of losing their common sense, there is no way to convince them by logical proof :-( In such a case, with strong management support you might still succeed to push the idea through, but without that, there is no hope. Jun 30, 2011 at 11:09
  • 3
    You 'break' tests often by changing the interface, removing Classes, etc. When you do 'test first', you dont experience this as 'breaking', but as the first step 'Red'. But when you're in the 'Just add some lines until it works'-mindset, the tests are only more lines. And when 'fixed' by someone like this, the tests tend to degrade in usefulness, until they really serve no purpose. Selffulfilling prophecy 4tw!
    – keppla
    Jun 30, 2011 at 15:00
8

All code must be tested. There is no choice about that.

Untested code is useless: it cannot be trusted to do anything.

You can write unit tests or you can hope to write higher-level integration tests or acceptance tests.

Unit tests are easier to write, easier to debug, and easier to manage.

Integration tests are based on assumptions that the units actually work. Without unit tests, how do you know the units work? How can you debug an integration test if you don't know the individual units being integrated actually work?

Acceptance tests, similarly, depend on all the pieces and parts working. How can you know the pieces and parts work without having a suite of unit tests?

Testing is mandatory. All higher-level tests depend on the units actually working.

That makes unit testing a logical necessity. There is no other choice.

5
  • 1
    There is no other choice ... if you care about quality. Most software organizations don't genuinely care about quality (in so far that they refuse to ship software known to be broken). Jun 30, 2011 at 14:36
  • @Joeri Sebrechts: It's not a quality issue. It's a "is it done" issue. Even bad software needs to be tested to prove that it does something -- anything. Simple logic dictates that there must be a test to prove it works. Even a bad test is a test. Simple logic dictates that a test of the whole must include tests of the parts. Unit testing is simply required by logic, nothing else. Not quality considerations, but by definition of "done".
    – S.Lott
    Jul 1, 2011 at 11:37
  • 1
    Using the software is a test of itself. Plenty of organisations are happy to make the user do the testing. I'm not saying that's a good thing, but it's the way it is. You can indeed choose to not test before delivery, offloading testing to the end user. You can, but you shouldn't. Jul 3, 2011 at 10:13
  • 1
    @Joeri Sebrechts: Actually, you can't. You cannot turn software -- at random -- over to a user for acceptance testing. You must have run the software at least once to be sure it appeared to work. That's a test -- a bad test -- but a test. Logically, that bad test incorporated bad unit tests. They existed, even though they were very bad.
    – S.Lott
    Jul 4, 2011 at 4:13
  • your definition of a unit test is useless for this discussion. A unit test is a codified test and that isn't neccessary at all to ship software. (but oc good to do in many circumstances)
    – Martin Ba
    Jul 26, 2012 at 7:12
7

I tend to side with your colleagues, but only up to a point.

The issue with unit tests is that they're frequently and mindlessly written on trivial cases where cursory investigation of the code reveals that it'll work no matter what. For instance:

def add(x, y)
  x + y
end

Along with a dozen tests to make sure that the addition will indeed work for arbitrarily chosen use-cases. Duh...

The general premise behind unit testing is: if your code contains no bugs, that's because you haven't tested enough. Now, when to write proper unit tests. Answers:

  1. When you're testing
  2. When you're debugging
  3. As you're developing really tricky things

Let's go through each one, supposing you're developing some kind fo web app.

You write some code to new functionality, and it should work reasonably well by now. You then reach out for your browser and verify that it works by testing more intensively, right? Bzzzt!... Wrong answer. You write a unit test. If you don't do so now, you probably never will. And this is one of the places where unit tests work very well: to test high level functionality.

You then discover a bug (who never misses any?). This brings us to point two. You dive back into the code and start following the steps. As you do, write the unit tests at key break points where having consistent and correct data is crucial.

Last point is the other way around. You're designing some hairy functionality that involves loads of meta-programming. It quickly spawns a decision tree with thousands of potential scenarios, and you need to make sure that each and every last one of them works. When writing such things, a simple looking change here or there can have unimaginable consequences further down the food chain. Say, you're designing an MPTT implementation using SQL triggers - so that it can work with multiple-row statements.

In this kind of thorny environment, you'll typically want to highly automate your tests. So you write scripts to automate the generation of test data, and run a boat load of unit tests on this test data. One crucial thing to not lose track of as you do this, is that you also need to write unit tests for your unit test generator.

Bottom line: unit tests, definitely yes. But spare yourself the ones on basic functionality - until you actually need them for debugging, or making sure some hairy functionality works properly (including, in the latter case, the tests themselves).

2
  • As Kent Beck put it: "test everything that could possibly break." Jun 30, 2011 at 8:50
  • 1
    Agreeing wholeheartedly with him. My main hope is that the OP will take note of: don't forget to test the tests where applicable. :-) Jun 30, 2011 at 8:54
6

I use unit tests for everything I write, and I don't let any untested code pass review without a test case. (But I lack a coverage tool, so I have to reason about test coverage. One thing at a time...)

It's quite simple: if you can't demonstrate that your code works (and what better way than an executable specification in the form of a test?) then it does not work.

This gives me:

  • Peace of mind: my CI server tells me my entire codebase works.
  • The ability to improve my code: I can restructure my code - refactor it - knowing that after the restructuring I have broken nothing.
4
  • 2
    I'd add one very important caveat to that - unit testing alone will not guarantee that "your entire codebase works". It will ensure that all units of code work in isolation, but there's still a huge possibility for integration errors, errors in visual presentation of data, etc. Jun 30, 2011 at 13:13
  • Unless your environment goes by " if you can't demonstrate that your code doesn't work then it does work." :p
    – Davy8
    Jun 30, 2011 at 22:08
  • @Ryan: Of course your integration tests should drive the entire project. The question was about unit tests so I talked about unit tests, but of course you should demonstrate the need for a piece of code by having a failing integration test. And of course you should have a CI server automatically deploying your codebase and running all testsl Jul 1, 2011 at 10:48
  • @Davy8: In which case you have more serious problems that "do unit tests work?". Jul 1, 2011 at 10:48
4

Unit testing is a discipline. Because it is not necessary for code to work, it is often discarded.

It is, however, a proven method that leads to robust and less buggy code. I don't think I need to explain the benefits to you, as you already mentioned some. If you look at some leading open source projects, whether they are javascript libraries, full-stack frameworks, or single-purpose applications, they all use unit testing.

I suspect your colleagues that recommend against using it are not programmers. If they are, well let's just say there are not many people I value higher than people like Martin Fowler or Kent Beck ;).

Like most best practices, the benefits are long-term, you need to invest some time in it. You could write a long script of procedural code, but we all know that you wish you had written it in object-oriented style when you need to refactor it after 2 years.

The same goes for unit testing. If you write unit tests for critical parts in your application, you can make sure it works as intended, all the time, even after refactoring code. Maybe you code so well that you never have a regression bug. But if you don't, or a new programmer joins your team, you want to make sure bugs of the past don't happen again. I think your boss would be happy with that also, as opposed to having to file a bug report he thought was solved half a year ago.

3

Unit-tests require unit-test friendly language, unit-test friendly design and unit-test friendly problem.

C++, multithreaded design and GUI will be a nightmare, and the problem coverage will be appalling.

.NET, reusable single-threaded dll assembly, pure computational logic will be a breeze.

Is it worth it, I can't really say.

Do you refactor much? Do you see "stupid bugs" like "off by one" in your code much?

Whatever you do, don't be that guy that criticizes others by "you don't have unit-tests, therefore you suck". If tests help you, go for it, if not, it's not like they are a silver bullet.

8
  • 1
    Why would multithreaded design be a nightmare? Design for synchronisation points, and test there. Jun 30, 2011 at 9:47
  • 1
    Because timing depends on moon phases, CPU clock, trashing and pretty much anything. On one check-in the test might fail, if you're lucky, on 1000 others it wont.
    – Coder
    Jun 30, 2011 at 9:55
  • 1
    I've written test-driven multithreaded stuff. It's most definitely doable. Jun 30, 2011 at 10:25
  • 4
    rubbish, you can unit test in any language.
    – jk.
    Jun 30, 2011 at 11:17
  • 1
    It's hard writing unit tests for GUI interactions, but that's why we isolated anything (our engine code) not to do with the GUI. It helped us make a good decision to separate the engine from the GUI, plus, the Unit Tests act as our interface in lieu of the GUI, asking and providing the information we would normally need.
    – Chance
    Dec 15, 2011 at 20:27
3

I want to, but I've had the misfortune of almost entirely working at companies that just don't care about quality, and are more focused with throwing together garbage that kinda sorta maybe-if-you-squint works. I've mentioned unit tests and I get a "Huh?" kind of look (the same look I get when I mention the SOLID principles, or ORMs, or design patterns beyond "class with all static methods", or following the .NET naming conventions). I've only ever been at one company that understood that stuff, and unfortunately it was a short-term contract and the department got trimmed to cut costs so there wasn't enough time to really learn.

I have dabbled in unit tests in throwaway dummy projects, but I haven't really gotten my head around full-blown TDD/BDD, and not having a mentor who can help with it makes it that much harder to do properly.

2

We use them. One big benefit we get is our unit test framework checks for memory leaks and allocation failures. Sometimes the problem is in the unit test code itself, but often it's in the production code.

It's a great way to find those things that be missed in a code review.

Unit tests (should) also run very quickly and can run as part of your continuous integration after every single commit, and also by developers before committing. We have over 1,000 which take less than 20 seconds to run.

Compare with with 40+ minutes for system-level automation tests, and hours/days for manual testing. Of course, unit tests are not the only testing you should be doing, but at a fine grained code-level they can help find bugs and test those edge cases that system/integration testing cannot touch. Our code must be tested with over 75% decision coverage and with 100% function coverage, which would be very hard to achieve without unit tests.

2

I would say unit tests do add value, but I am not a big TDD disciple. I find the most benefit with unit tests when I do calc engines, or any kind of engine really and then utility classes, unit tests are very helpful then.

I prefer automated integration tests for the more data dependant blocks, but it all depends, I am in the data business, so many errors are more data related in our environment than anything else, and for that the automated integration tests work well. Checking the data before and after and generating exception reports from those tests.

So unit tests really do add value, especially if the unit you are testing can be supplied with all the inputs it needs and work all on its own with the given input. Again in my case the calc engines and utility classes are perfect examples of these.

2

Costs: Slower to code, Learning Curve, Developer Inertia

Benefits: Generally I found myself writing better code when I tested first - SOLID wise...

But IMHO, the largest benefit I think, is reliability in larger systems that change frequently. Good unit testing will save your butt (did mine) when you release multiple versions and can eliminate a big chunk of costs associated with manual QA processes. It will not guarantee bug free code of course but it will catch some hard-to-predict ones. In large complex systems, this can be a very large benefit indeed.

0

I do some unit test at work when I get the chance, and I try to convince my colleagues to do the same.

I'm not religious about it, but experience shows that utility and business methods that have been unit testes are very robust and tend to have few or no bugs showing up during the testing phase, which in the end reduces costs on the project.

0

Where I saw the real advantage to coding Unit Tests and making unit test execution part of the daily build process was on a project with 30+ developers working on 3 different continents. Where the unit tests really shined was when someone would subtly change a class they maintained without understanding how people were using that class. Instead of waiting for code to hit the testing group, there was immediate feedback when other people's unit tests started failing as a result of the change. [That's why all unit tests need to be routinely run, not just the ones you wrote for your code.]

My guidelines for unit testing are:

  1. Test every condition you can think of for your code, including bad inputs, boundaries, etc.
  2. All Unit Tests for All Modules should be executed regularly (i.e. as part of the nightly builds)
  3. Check the Unit Test results!
  4. If someone finds a bug in your code that got past your tests, write a new test for it!
0

I have recently learned TDD and find that it is very useful. It gives greater confidence that my code is behaving as I expect. Along with making any re-factoring that I wish to do easier. Whenever a bug is found you can ensure that through the testing that it won't show up again.

The hardest part is writing good unit tests.

It is a good benchmark for your code.

Not the answer you're looking for? Browse other questions tagged or ask your own question.