Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

My team is considering a department goal of doing Test Driven Development next (fiscal) year.

TDD sounds good and my manager is on board. The only problem is measurement.

Our company rules require that department goals be measurable.

Is there any (sane) way to measure if a team is doing TDD?

Note: We had a goal for writing unit tests already, so a code coverage goal will not work. We write the unit tests but we do it after the fact, not before.

Is there a way to measure that TDD is being done?

(NOTE: We use Visual Studio 2013 and TFS in case it matters)

share|improve this question
    
Junior Developper here: Maybe set steps with your manager. such as 1) write tests depending solely on specifications. 2)the coordinator/Project Manager checks that they comply with the specifications; 3) Developper can start. That may be slow a first glance, at least, it will show which specifications are poor or well-written. –  Fabinout Feb 5 at 0:01
4  
Your question brings up an interesting point; does TDD (rigid) make any measurable, objective difference in development? So many development methods are untested in objective terms. You may actually get some data. If the use of TDD is un-measurable... –  Frank Hileman Feb 5 at 0:30
    
If it is just to satisfy the bean counters: commit the tests before and separately from the code. The earlier commit times are an indication that the tests were likely written first. –  Bart van Ingen Schenau Feb 5 at 8:20
    
Doesn't measure whether you are "doing TDD", but a good articles on test metrics nonetheless: Selecting Developer Testing Metrics –  Marjan Venema Feb 5 at 11:17

8 Answers 8

Just like a unit test should test behaviour not implementation your department should measure the anticipated result of the policy, not implementation of the policy.

Your department wants to move to TDD. Why? What is it you're hoping to achieve through TDD? Fewer bugs raised that are regression related? Faster velocity on cases? Then you should measure these things instead of trying to find a way to measure that everyone is using TDD.

If the hoped for results of your push to TDD isn't something externally measurable, such as a more enjoyable developer experience and improved flow (one of the things I like about TDD), then you shouldn't need to change your testing at all because you'd be "refactoring" the department.

Instead I suggest just asking for feedback from the developers on how they think TDD is going. If they don't 'buy-in' to the TDD concept then it doesn't matter how you measure it - it will fail.

share|improve this answer
    
I agree you should measure the results of those policies, and perhaps the results should be the goals but you should also measure the policies! If you do, you can see whether the policies do anything. If velocity on cases is increasing, but your TDD measure is not, it is clear that you have some other variable changing and should look for it. –  Alan Shutko Feb 5 at 3:06
    
We (the developers) want to do TDD. It is just the "bean counters" that need a hard measurable thing. "Fewer bugs" is too fuzzy as way too many things can contribute to that. –  Vaccano Feb 5 at 3:26
4  
Did you set the goal of having a friendly and welcoming work environment? good luck measuring that.. there are times when you have to talk sense into bean counters for their own sake. I agree though, that TDD as a goal may not be a good choice, as it is just a means to a more goaly end. –  Frank Feb 5 at 7:04
2  
@AlanShutko I think it should be left to the developers to decide amongst themselves how to organise to achieve their work. The output of their development process is what matters (to anyone outside the team) - if the dev team make 5 changes to the development process and productivity and quality goes up as a result does it matter which of the 5 changes caused the improvement? Development workflow should be a process of continuous improvement that is left entirely to the dev team to control, with management just caring about quality of output. –  combinatorics Feb 5 at 8:46
1  
@combinatorics, it's interesting you'd say it wouldn't matter which of the five changes made the difference, because you would never say "I made five changes to the source code, and it really doesn't matter which one fixed the bug." –  Alan Shutko Feb 5 at 16:23

Since you are going to measure unit test separately (code coverage, decision coverage, etc.), let me add a couple more tweaks to the process.

  1. All check-ins must go through code review.
  2. Code reviews should be trackable, i.e, should not be an over-the-shoulder review or like pair programming. You should be able to retrieve the code review comments, iterations and such details at a later time. TFS lets you do this.
  3. When signing-off on a code review, the sign-off comments can a predetermined piece of text that tells TDD was followed. e.g.: "TDD FOLLOWED". This way you know which check-ins followed TDD, and at the end of the year, you can invent/calculate more metrics like percentage of LOC with TDD.

HOWEVER, I would not judge the value/effectiveness of TDD purely based on these metrics. I think the industry has evolved enough that the majority understands the value of TDD.

share|improve this answer
    
the person doing the reviews can't know whether you wrote the test before or after the code it's testing, only that both exist. And oh, most teams will use checked in code for their code reviews. –  jwenting Feb 6 at 12:01

Since you want to switch from a test-second development to test driven design, you can best measure that by looking for proof that the tests were written before program was changed.

Teach your team to commit their failing tests first, so they can be measured, and then write the code to pass the test. You may need to either manually inspect the repository history, or write a custom tool to do such for you.

(You'll wind up treating it like a check-off for each project, so if check-in-first is needed you could always just have it be part of the reported information, and trust the programmers / project manager. i.e., "were the tests written before the code?")

share|improve this answer
1  
I don't think this will work in many development set ups. Checking in a failing test will mean a continuous integration server either won't be able to produce a new build any time the branch has development in progress or it will be able to produce a new build and you'll have to guess at how many tests were expected to fail in the process. Also, if I pull a copy of the code after you've checked in the failing test and before you've checked in your code change I will open the project in my IDE and the first thing I see is a mysterious failing test case which I then need to investigate. –  combinatorics Feb 5 at 22:24
    
This is a good idea from the metrics standpoint (allows measuring TDD). However it would prove too disruptive to my team for development. I would need some way to isolate the "broken" checkins. –  Vaccano Feb 5 at 22:30
1  
Separating out your "development" and "build" branches will alleviate those concerns. (You are using a distributed version control system, like git, right?) –  DougM Feb 5 at 22:32
    
@DougM Sadly we're not, and I'm not denying it could work in some setups. I've had a look around stack exchange just now to see what the community thinks about checking in failing tests and it seems to be more common than I thought (although generally frowned upon?). I'm definitely willing to admit my unease at checking in broken tests might be more due to my specific experience and setup than any inherent problem with it. –  combinatorics Feb 5 at 22:59
    
If I checked in failing tests in my job I'd have 3 angry managers on the side of my desk within half an hour because the automated build for the development environment failed. Tests are committed with the source they're testing. And THAT might be tracable from log analysis of your source control system logs if you have strict naming conventions. –  jwenting Feb 6 at 11:59

If a team truly wants to do TDD, they you just need to do some minimum tracking to satisfy the rule. Check in your tests first and then check in the code and after comparing the initial dates, you have some semblance of proof.

To have another layer of approval, make this a part of the code review where someone can verify these are meaningful tests. It's not that hard to write a test that fails without even referencing the code it is suppose to test. It's just not in the spirit of what you're trying to do.

During some semblance of pair-programming, everyone can "sign-off" that TDD was going on.

You may not feel the need to take things this far, but you have to be prepared to serve the powers that be.

Personally, I think TDD should be used and evaluated to solve some larger problem. If releasing fewer bugs on your next upgrade is a concern because it lowers customer satisfaction, you can do a survey and ask customers: On a scale of 1-10 (10 being most painful) how much more painful was our latest upgrade (built on TDD) compared to the previous one? It's good enough for pain medication drug trials, is should be good enough for software.

share|improve this answer

TDD is not something a team commits to doing. It's only a means to an end. Objectives can be writing more maintainable code, decreasing the number of defects, delivering faster, or improving test coverage, but not "doing TDD".

TDD is not a silver bullet. Measuring "if the team is doing TDD" does not guarantee success like making sure they drink their magic potion every day. Besides, demanding that the team produce indicators showing that they are practising a technique that the team itself asked for in the first place is a bit... absurd. Or command-and-control fundamentalistic.

Managers and decision makers shouldn't demand control over how code is produced at a micro level (a typical red/green/refactor cycle is the order of a few minutes). Since when do bean counters care about how the beans were harvested ?

share|improve this answer

It's a stupid request proving the idiocy of many managers/accountants/cto etc who don't understand that software development is a kinda unique process in itself (so no, it doesn't matter that Toyota uses a specific process, you're not building cars)

Anyway, the only things that matters to be measured are RESULTS. Do you deliver higher quality software? Do you deliver it faster? Does the current method of developing (TDD or not) yields at least equal or better results with lower cost (time and money)?

It simply boils down to cost and results. You want better results with the same or lower costs or at least the same result with less costs. It's the ratio between the final result and the bottom line that matters.

Your team has to show that the TDD approach improves productivity (maintainability -> easier changes -> faster results -> less time required) and/or product quality (less bugs -> less time spent debugging -> reliable product | less people pissed off -> happier stakeholders).

I know that it takes more time to think and write tests but software devel doesn't stop when the first release is out. And even beancounters need to (forcefuly) understand that software development has its quirks, it's not an exact science and if they can do a better job, let them do it. You don't build houses how you build cars or airplanes, a cupcake business or cargo services. Why should you build software this way?

Edit: It might be news for some people but a lot of things simply CAN'T be measured reliably. You can pretend you're measuring it and come up with bullshit metrics but that's all. Marketing is one of the domains greatly affected by this plague.

share|improve this answer
    
Just measuring results is a poor metric to gauge the success of one of several process improvements (we don't do just one a year). If it all about results and you tried to improve 3 things, how do you know which ones helped? Or even which ones were done? –  Vaccano Feb 5 at 22:32
    
I've already said that some things can't be really measured and personally I wouldn't consider to convert the software devel to numbers. If 1-2 good developers leave and only some junior remain, it doesn't matter that your process uses TDD,DDD, whatever DD. The results will suffer. And someone will come up with yet another stupid formula for 'better' metrics, when the reality is much simpler: it's about developer skill and experience, not about assembly line process. How can you measure dev skill or exp? You can't, unless you consider 7 years of C# to be a meaningful metric. –  MikeSW Feb 5 at 22:40

I'm not sure that you'll need a different measure at all. You say:

We had a goal for writing unit tests already....

... presumably you are measuring this in some way. The fact that you now want to write tests before the code should have no effect on your existing metrics. After all, there may have been times when, (for example) to get a better understanding of the problem, someone in the team did write a test before the code. So in that case they were already practising TDD; they just didn't call it that, and management didn't care.

The existing goal is to have unit tests; you will still have unit tests after adopting TDD. It's simply a non-issue from a bean-counter perspective. As others have said, TDD in itself is not a goal.

share|improve this answer

TDD should generate at least 1 unit test per class. For every class in a project there should be at least 1 unit test that was written for that class, and it tests just that class.

Unit test to class ratios, code coverage, etc.. etc.. seem like things that give a meaningful measurement, but they don't.

What you want to measure is the complexity of classes, and how well those classes are tested. The higher the complexity of a class the greater the risk that class introduces into a system.

You can have 100% test coverage on a highly complex class, but that doesn't mean you've tested it well.

Complexity is the measurement of how many logical decision pathways there are in code. You can have a chunk of code that generates 1 million possible decision combinations and have 100% code coverage. It doesn't mean you tested all 1 million cases.

The other issue is how much of the system uses these complex class?

So this means some unit tests are more important than others, and it's those tests you want done very well.

This is something that is very hard to measure...

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.