Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

To start I don't think this is a repeat of other questions on unit testing. What I'm looking for help with is articulating its value to a team of programmers, analysts, managers and testers. By automated tests, I don't think I need to make a distinction between unit tests (e.g. JUnit), BDD (e.g. JBehave, Fitness) and UI (Selenium , Watir) because I think they all provide similar value (but feel free to write an answer that disagrees :))

The following is a list I've identified, I'm looking for answers that help expand or refine:

  1. Time/Cost Savings: writing automated tests can take more time than written test cases. However, considering tests are run multiple times, the marginal work (i.e. cost/time) to execute automated tests is several orders of magnitude less. That the automated tests are cheap to run facilitates changing the system over time.
  2. Documentation: there is no truer way to know how a system works than its tests. Any other documentation is usually out of date the moment its written, but tests (at least those that pass) reveal how things actually work. This is true for both end-user AND API documentation.
  3. Code Quality: test writing forces you to:
    • consider clients because tests are a client
    • breaks dependencies where making code testable often means figuring out how to make that code not require some other large system be available
share|improve this question

migrated from stackoverflow.com Jul 6 '11 at 11:49

This question came from our site for professional and enthusiast programmers.

10 Answers 10

up vote 16 down vote accepted

A few of my thoughts:

  1. Be honest that writing automated tests will take more time. If you're doing unit level TDD (which I would recommend as a starting point if you're going to invest in automated testing), you can expect about 30% extra time needed to code a feature. The key here is explaining that this extra 30% (which is probably higher than 30% in the beginning as your team learns how to write good tests) is an investment built to save costs over time. As least with unit level TDD, the design of your system is loosely coupled and highly cohesive, which makes your system adaptable to change over time. New requirements and unexpected bugs always require changes to your system, so keeping your system in a state where change is easy (from the nice design that comes from TDD) and safe (the set of automated tests that continually verify your system) means that it will cost less money to make those changes.
  2. There's lots of debate about the value of Acceptance level and UI level tests given the amount of time it takes to write these tests, how long it takes to run them, and how much maintenance they require. I'd recommend reading this article by James Shore about this.
  3. In the world of automated testing, there are good ways and bad ways to do it. If you are pitching automated testing to your management, I would pitch alongside it how you're planning on getting your team trained in writing good tests. The Art of Unit Testing by Roy Osherove, Working Effectively With Legacy Code by Michael Feathers, and The Art of Agile Development by James Shore are all great books that deal with these topics directly or indirectly. You should also look into some sort of coach or formal training as well. It's a big change.
  4. In terms of Business Value, #2 and #3 of your points above actually serve your first point, so I'd hammer home on point #1 and talk about how #2 and #3 serve that greater point. Documentation makes your system more understandable, which makes your team work faster. Code Quality makes your system adaptable to change, which makes your team work faster. For business people, it's all about maximizing the flow of value from the time an idea is pitched to the time the idea is delivered as working software.
share|improve this answer
1  
+1 good answer. Interesting link to James Shore article. I would add The Clean Coder by Robert Martin to your book list. I think developer created UI tests should cover happy paths whilst QA (if it exists) writes exceptions. Unit tests should really address the exceptional cases. –  orangepips Jun 30 '11 at 16:24
    
@orangepips - Thanks for the book recommendation. One downside to this UI tests being the happy path and then unit tests covering exceptions is that it's more difficult to write those unit tests if you aren't doing unit testing for everything. Unit testing helps drive the testability of your app by keeping coupling low while UI tests don't require that the code underneath is loosely coupled. –  Brian Geihsler Jun 30 '11 at 18:17
    
meant to write Unit Tests should cover everything. –  orangepips Jun 30 '11 at 19:36
1  
@orangepips - I disagree. "QA level"/Acceptance tests should test everything that matters to the user.. i.e. happy paths and alternate scenarios. Unit tests frequently use mocks, stubs and fakes... which means there is a possibility that the happy path unit test passes but when all the components are brought together, the happy path end-to-end test may fail. It is too much of a chance to be left to fate. –  Gishu Jul 1 '11 at 4:41
2  
@orangepips - My objection was related to the QA/Dev Exceptions/Happy divide. Unit tests exist to ensure that you're building it right. QA/Acceptance tests exist to ensure that you're building the right system. So all scenarios that are relevant to the business (e.g. the credit card has expired) should be tested by QA before they brand it ready to ship. I recommend automation of acceptance tests - Automate the tedious, routine stuff 80%+. Top that off with some imaginative non-scripted manual testing. –  Gishu Jul 4 '11 at 13:08

Mostly (depending on your test coverage) bug-free code, and I would say that one of the biggest arguments is when you say to your manager that you can write a test for a discovered bug, ensuring you will always know in the future if that bug comes back :)

My opinion is that unit / integration tests are most important, while if you apply some UI pattern like MVC it is enough for most projects. I usually test all actions on my controllers / presenters, and leave databinding to the Views.

Of course, automated testing does not substitute good old point and click adventuring around your application trying to figure out wildest things your user could do.

There is also a point of Continuous Integration.

One more thing - one must strive that code quality leads to product quality, business value, and maintainability - otherwise there is no point in doing it.

share|improve this answer
    
+1 for Continous Integration from a technical point of view. But, I'm not sure how I see your suggestions facilitating a conversation with non-technical staff (e.g. analysts). Also, what are your thoughts about validating across multiple browsers and OSes? –  orangepips Jun 30 '11 at 13:14
    
Well, I can just tell my side from the standpoint of the developer, about the analysts - I don't really fully understand their roles in really big projects - so no real advice there. About the cross-browser cross-OS tests, didn't have chance to do those either. –  Denis Biondic Jun 30 '11 at 13:45

One thing of definite value is that automate tests can be run continuously; such as every hour on a rebuild or similar. Doing this forces any bugs or regressions out into the open rapidly within hours or days of a programmer working on the offending code, this makes context switching much easier. The second benefit to continuous testing is that it forces you to keep you tests in a working state; nothing is more tedious than spending the first week of a test cycle fixing all of the out of date tests. If you can automate them, you can run them any time & by running them regularly you can catch bugs in your tests or code quickly.

share|improve this answer

Ease of refactoring is a big factor here. Having good coverage by nice and READABLE (!!!) unit test, you can refactor your system without being nervous about compromising existing functionality.

share|improve this answer
    
does this differ from my point #1? –  orangepips Jun 30 '11 at 16:25
    
@orangepips: No, I missed that part. Sorry :o) Still, it is important to emphasize –  Morten Jun 30 '11 at 18:18

Test Expense

Once an automatic test is written, it can be run by a computer at the cost of a few joules. The equivalent manual test requires a person on payroll working down a list of instructions.

Test Reliability

The computer can be trusted to faithfully execute the same test procedure, every time. The human is apt to make mistakes and get lazy.

The computer's testing failure modes are also much more readily apparent - it crashed (test reports stop appearing), it had a bit error that caused a false test result (run a deterministic test again, and the result differs). If a human misses a step and checks off the "OK", how can we tell?

Test Durability

An automated test has to be a concrete artifact (e.g. a piece of code) in order to run, and is naturally included with the other software development artifacts - the source repository. A manual test may be developed on a sheet of note paper by a tester, and never formalized. The business is more likely to need processes in place to ensure that doesn't happen.

Test Value

The computer can be programmed to output test results in a consistent, easily analyzed form. The person is either doing data entry to generate the same, or is recording free-form notes that require an analyst, developer, or manager to digest.

share|improve this answer
    
+1 for the notion of reporting and for referencing joules. –  orangepips Jul 1 '11 at 0:50
    
"The computer can be trusted to faithfully execute the same test procedure, every time" It is worth remembering that some errors are found by people doing things in an unexpected way. Quite often a different tester will perform the same set of instructions in a different way. This is a good thing as it increases test coverage so although test automation saves time and is a great way to check for expected failures & regressions it cannot totally replace human testing. –  TafT Jul 1 '11 at 13:48
    
In that case, it seems preferable to give the human testers general lists of areas to explore and things to try, and not detailed instructions that they should faithfully repeat. –  Novelocrat Jul 2 '11 at 0:06
3  
@TafT: only the poor or foolhardy go without manual testing, but the highest value manual testing I believe is exploratory rather than wrote in nature. Thus the push to automate that which can be. –  orangepips Jul 2 '11 at 7:13

I think you should lead with the magic points of "lower Cost" and "more Features/unit time" / smaller cycle-time.

However before making a case, I'd advise to reflect on your situation. Your question led me to write a blog post on the potential cons of automated testing.

share|improve this answer
    
+1 for a good blog post, albeit those points are something that would be well raised here. It strikes me the primary concern is not having programmers who just go through the motions. To that end then, how do you suggest promoting quality or least avoiding an atmosphere that prevents it? –  orangepips Jul 1 '11 at 8:42
    
good link. Maturing any software process takes a lot of work. I think the important corollary is also reducing turnover so you have enough people with an organization memory and trust to move something like this forward. –  orangepips Jul 1 '11 at 13:01
    
+1 for an excelent blog post!! –  Morten Jul 4 '11 at 12:14

You have to sell the concept – you need to avoid telling them it will improve the code. If they have any investment in the code that will immediately put them anti you/auto testing. If they are any good they will also understand GIGO so will not understand why you think it doesn't apply.

I'd leave selling it as documentation aspect out as well, things like Fitnesse can do it well, but until they experienced it might be hard to visualise.

Areas I think that might have luck selling it on

  1. Unit Tests can take the place of many developer harnesses – where you create application just to get at area to debug/test without going through all the login/menus.

  2. Tests allow you to set-up and easily repeat problems situations – without spending lots of time setting up test data (especially using a decent mocking system)

  3. As you build up suites of BDD & UI tests – you get a much quicker response if there are simple breaks than waiting for next time the tester looks at it

  4. BDD & UI tests can avoid you repeatedly pressing buttons to check all aspects that might have been affected by your change and save you having to remember every areas.

  5. Automatic builds often highlight when someone has forgotten to check in code

  6. Tests help you avoid bugs reappearing.

  7. Unit Tests and decent mocking will mean less inter-linked code and will be easier to resolved

Remember you are trying to sell it, not convert them to a religion – so accept small steps and try not to get them anti-you. It will also take time for them to adjust and learn to write good tests.

share|improve this answer
    
+1 for the religion comment. I think there's a matter of identifying what's worthwhile to write automated tests for and clearly the answer is not everything. OTO, I think we're better off having at least some automated testing. Perhaps the real key is acknowledging that at least in my organization the SDLC bottleneck is QA. So my own effort is directed at smoothing out that effort curve by having development assume some of that responsibility. –  orangepips Jul 1 '11 at 8:49
    
In relation to number 3) this allows you to build up stats and form reports; visibly can be a big selling point. This week introducing feature X caused 10 tests to fail which we detected in Y time thanks to automated testing is a nice "win" for a project, also helps document the risks of introducing new features in the future. –  TafT Jul 1 '11 at 13:50

Someone must beleive there is a problem before they will accept a proposed solution to that problem.

Automated tests can save bug-fixing costs, so if your colleagues don't beleive that bug-fixing costs are sizeable or excessive, they will be hard to convince. If those costs are high or excessive, but people do not believe they are, you might first have to get some convincing data about those costs.

share|improve this answer
1  
so where do you think that information should come from? –  orangepips Jul 5 '11 at 14:50

What businesses love is increasing value and lowering cost. You have to explain how automated testing will increase value since it does add an extra cost.

If your code is modular it will be possible to re-use it. Which means that the tests do not have to written all over again and you can just work on top of that existing code.

If there are legacy projects, automated testing makes it much easier to refactor. The technical debt has to be paid down at some point.

The documentation argument you provide isn't very good. The difference between keeping tests up to date and documentation up to date is just habit.

share|improve this answer
    
In my experience code reuse is an emergent quality of software not planned. In other words it wasn't until I rewrote the same thing the third, fourth or fifth time that I truly understood how to make it reusable. So I think managers have often felt burned by the notion from programmers of "give me more time to build it right and that will lead to cost savings" because in practice I find this to be a generally false approach. –  orangepips Apr 29 '13 at 13:43

"What I'm looking for help with is articulating its value to a team of programmers, analysts, managers and testers. By automated tests, I don't think I need to make a distinction between unit tests (e.g. JUnit), BDD (e.g. JBehave, Fitness) and UI (Selenium , Watir) because I think they all provide similar value (but feel free to write an answer that disagrees :))"

OK I'll take that challenge ;)

I mostly work with programmers and QA and my tools are ruby, rails, testunit, rspec, jasmine and selenium.

The BDD/TDD tools of rspec and testunit are part of programming. You don't break them out and talk about them separately to management, you don't put them off due to lack of time, you include them in all your time estimates. If really pushed asked how much time people have for you to explain computer science and programming to them. I don't use these tests for the front end

GUI/UI/Jasmine/Selenium. These are different. I have these done by QA folks who have programmer backgrounds. We make sure the tests are written to be as robust as possible and based on content not layout. The (possibly new) cost of this should be explained as a shifted cost. Instead of paying with broken software, lost customers and expensive fixes later, you pay a lot less (relatively) now with a few simple practices.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.