Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

This question already has an answer here:

I asked a question on lines of code per hour and got torn a new one. So my matured follow-up question is this:

If not lines of code, then what is a good metric by which to measure (by the hour/day/unit-of-time) the effectiveness of remote programmers?

share|improve this question
21  
How do you measure the effectiveness of non-remote programmers? Why do you think it needs to be different? –  Jon Hopkins Dec 15 '10 at 9:15
15  
Measuring "lines of code removed" as a positive is a better metric than "lines of code added". –  Christoffer Hammarström Dec 15 '10 at 14:03
2  
@Chrstoffer: For fun, that's how I tend to measure how productive I was on a given task at my day job. I love it when I can reduce the code block I'm working on by 33% and put in more functionality. –  Jeremy Heiler Dec 15 '10 at 14:10
1  
@Wonko the sane: I think the question applies to both camps equally. Either is not working in the office and this worker needs metrics placed on him/her. –  Chris Dec 15 '10 at 19:18
1  
@Jon, true...it may be the same. I just meant to clarify that I was without a mechanism to actually talk to them one-on-one in a room. –  mmmmm Dec 15 '10 at 21:07
show 4 more comments

marked as duplicate by gnat, Jalayn, Yusubov, MichaelT, Joris Timmermans May 22 '13 at 8:36

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

11 Answers

up vote 70 down vote accepted

In 16 years I've never actually found a workable metric of the sort you're looking for.

Essentially to be useful anything would need to be measurable, representative and ungameable (that is the system can't be played by clever developers). There are simply too many variables within software development to make it measurable as piece work in this way.

The closest you get is progress against estimates - that is how many tasks are they completing within the agreed estimates. The trick here is (a) getting good and fair estimates and (b) understanding where estimates have been exceeded for good reasons for which the developer can not / should not be blamed (that is something was genuinely more complex than anticipated). Ultimately if you push developers too hard you're likely to find estimates gradually creeping up to a level where they're always met not because of increased productivity but because of padded timescales.

Go the other way too much in terms of the estimates (reducing them to create pressure to deliver) and you create phoney deadlines which studies have shown don't increase productivity and are likely to have an impact on team morale (see Peopleware for more information).

But essentially I wonder if you're looking at a slightly false problem. Why are remote programmers different to other programmers when it comes to measuring productivity? How do you measure the productivity of non-remote programmers?

If it's about not trusting them to work remotely then that's basically a wider trust issue. If you don't trust them to work from home then you either need to establish that trust, not let them work from home, or find some way of satisfying yourself that they are indeed working when they're meant to be.

share|improve this answer
13  
+1 for pointing out trust issue. –  user1249 Dec 15 '10 at 9:39
6  
I don't think ungameable metrics exist. See for example: lesswrong.com/lw/1ws/the_importance_of_goodharts_law –  StasM Dec 15 '10 at 9:59
1  
Don't forget to account for code quality. Right now, I'm trying to get permission to rewrite some intern-written code that, in fact, met specifications. We had to special-case it out of certain warning diagnostics that are not allowed in the rest of our code, and now we want to change it some. –  David Thornley Dec 15 '10 at 14:48
2  
Also account for the fact that different code (and tasks) might have a different "par" for a given metric. If a developer is involved in deep changes to a core system that is convoluted and poorly documented then I would expect a high number of bugs from that developer. It's not necessarily that dev's fault that he generates an above average number of bugs and a below average line count. –  JohnL Dec 15 '10 at 19:14
add comment

The only metric I use is the amount of working software he produces for a given amount of money I invested.

Regardless its schedule, if she/he work remotely or not, number of pauses he/she takes, methodologies she/he uses or number of working hours.

By working software I mean:

List of features defined by user/customer that meets the required quality level

By amount of money:

How much the user/customer paid for the defined features + maintenance costs

So it has a direct on how it is built and quality of work produced but not bound to any source code line metrics.

share|improve this answer
11  
Though good luck quantifying the "amount" of working software... –  Jon Hopkins Dec 15 '10 at 9:05
19  
The notion of "amount" is totally ambiguous, as is the number of features. It only makes a little sense if you're able to define the "features" as similarly sized pieces (as in Scrum stories), but this is just pushing the ambiguity further. What is "similarly sized"? It's a story with the same amount of points. What's a story point? It's the developer's estimate of how long it takes to implement. Ta'da, the circular reasoning is complete! –  Joonas Pulakka Dec 15 '10 at 9:29
5  
I still don't think this meets the definition of "metric". Comparing a bullet list of features as having equivalent value isn't realistic--what if one feature is "web site should have a link to my email address in the footer" and another is "CRUD interfaces for each database table"? I think at some level you're attempting to quantify this "value" variable that is still basically a gut-feel thing. Not that I have anything against gut-feel (it's what drives most performance reviews), but it's not a METRIC. –  Dan Ray Dec 15 '10 at 13:25
2  
@adolf: It's exactly what I do. I use excel exactly like you, but instead of putting line of code of function points, I put completed user story points (with a well defined Definition Of Done). Completed story points per month is how I calculate the productivity of a team. I will never calculate their productivity by the amount of line of code, classes or even function point. I (me) don't see value in that. Please understand I'm not saying it's the way to go. It's my way to go. –  user2567 Dec 15 '10 at 13:41
1  
@Roopesh: time is bound to money. Time is money ;) –  user2567 Dec 15 '10 at 13:46
show 16 more comments

Metrics work best in factories, and programmers don't work on an assembly line.

I completely understand the desire to measure productivity.

But would you use the same metric for a family doctor and a heart surgeon? How about for Michelangelo painting the Sistine Chapel, and some guy in Mexico cranking out black velvet Elvis paintings?

Louis de Broglie wrote a doctoral thesis that was so short, the examiners were going to reject it - except de Broglie was a highly-placed aristocrat, and they needed a good excuse. So the examiners sent it to Einstein, who not only didn't reject it, he referred it to the Nobel committee, and de Broglie got the Nobel Prize in Physics for it five years later.

Numerical measures work best on work that's repetitive, like casting iron or screwing bolts on car doors. But if you're repeating code that's been done before, you don't need a programmer, you just need a copy-and-paste. Programming is fundamentally a creative discipline, and productivity depends entirely on what you're doing.

Some days, I crank out 1000 lines of code. Today, I'm going to be fixing coordinate geometry bugs, and the code might shrink. If I had to fix a bug in a Linux kernel driver, I might spend all day on debugging, and not write a line of new code.

Measuring programmer productivity is very, very, very subjective.

If you want to know if Joe is productive, find Sally and Ralph, who know what Joe's doing and are proficient in the same areas, and ask them.

The best numerical system I've ever seen has been Agile's planning poker points. That's just a fancy way of asking Joe and Sally and Ralph how hard they think Joe's upcoming job is likely to be. Then you can measure points-per-week productivity for each team member. But even then, it takes a while to calibrate a team's estimations, and the numbers are fuzzy and easily thrown off.

Many people want productivity estimates so they can do schedule planning. It's kind of the "plug it into MS Project, look at the critical path, and there's your ship date" theory. I have never, ever seen that work - there are just too many unknowns. If you want that, use Waterfall, design everything up front, don't permit any change orders, and be prepared to be disappointed anyway.

share|improve this answer
7  
They work on an assembler line. <tumbleweed.gif> –  adolf garlic Dec 23 '10 at 10:16
    
@adolf garlic: LOL! First laugh of the morning! –  Bob Murphy Dec 23 '10 at 16:56
2  
Perhaps calling programming a "creative discipline" is painting with too broad strokes: every job that requires a thinking cap is by your words then "creative". I agree with your point though, and try to get mathematicians to predict how long time it'll take them to crank out a proof of x, even within a well-traversed area. Or get them to quantify "work" done on that proof. We're indeed very far from the assembly line here. –  Steen Aug 30 '11 at 20:38
add comment

You need an experienced developer or teamleader (who is not associated with those remote programmers) to estimate how long some task may take, and the effectiveness is measured by comparing their required time against the estimates. To be sure that the estimates are good, you could randomly pick a few tasks and have them executed by an in-house team you have under control.

share|improve this answer
5  
+1 - The trick here is how you handle deadlines that are missed when they are. You need to make sure you really understand the reasons else you'll get developers looking to avoid the difficult tasks for fear they'll be penalised if it over runs - obviously what you actually want is developers, particularly your best developers, keen to work on these as they're the ones you really need nailed. –  Jon Hopkins Dec 16 '10 at 22:05
add comment

Another interesting way to measure productivity would be to count automatable tests reviewed by a manager per day. The developer gets:

  • a point for writing an automatable test (and passing review) and adding it to the regression test suite,
  • a point for making them pass, while not causing any other regression test failures.

The developer and manager can jointly improve the system by:

  • jointly agreeing on the important areas of development and testing
  • independently reviewing and running the test suite.
  • deciding not to build a feature that has limited business benefit but requires lots of development and testing required to deliver that feature. (The most productive line of code is one that you decided not to write because it delivers no business benefit).
  • partitioning the system into an architecture (such as model-view-controller) which facilitates incremental feature development without breaking the whole system.

The developer cannot game the metric because:

  • redundant tests will be blocked by manager review.
  • fine-grained tests may be blocked by manager review.
  • fine-grained tests will improve the quality of the system.

The manager cannot game the metric because:

  • rejecting too many tests will lead to developer attrition.
  • requesting too many tests will make it hard to refuse a later deadline.

The developer cannot screw the manager because

  • Each delivered unit of functionality with tests must also pass the regression suite. i.e. This forces the developer to develop carefully without breaking other code.
  • Any claim of work must be provable by passing new tests and regression tests.

The manager cannot screw the developer because.

  • Each requested unit of functionality must include key test cases, and an estimate of the number of test cases needed to finish the work.
  • It's hard to ask for an aggressive schedule and/or free overtime when you are obviously asking for a lot of work.

Another big benefit to the manager is that he can bring on new developers and know that they will not be able to deliver code that silently breaks the system (because the regression test suite catches that).

The big downside to the manager is that it forces him to admit that his system is more complex than it seems on the 1-page description of the feature. The other downside is that the transparency of this method will make it hard to blame developers for business failure.

share|improve this answer
add comment

It certainly is possible to devise all kinds of intricate metrics to evaluate performance, but at the end of the day a significant part of your judgment has to rely on subjectivity and input from people who are close to the codebase.

For example, it is quite possible for some team to crank out internally hideous unmaintainable slop at a very fast rate, and this might even meet the required deadline and specification. But is the technical debt accrued from that kind of working style worse than if the team had churned out something robust and maintainable but slipped the deadline a few weeks? It depends.

If the purpose of the question is to resolve some type of productivity problem, I would say that what the manager actually does to facilitate the work of the team is as or more important than any measuring technique used to evaluate the team. It is a two-way street. In other words, I am saying that metrics are fine but if you want more out of any team the ultimate question is whether or not the manager doing everything possible to ensure the team can be productive.

This goes far beyond writing a spec, finding a team, throwing the spec "over the wall" and clicking a stopwatch.

share|improve this answer
5  
I worked at a place where you could give person A a task, and they'd solve it in two days by adding 150 lines of bug-free code that completely met the requirements. You could give the same task to person B, and they'd solve it in one day by adding 3000 lines of code that failed half the unit tests, largely consisted of the same code copied-and-pasted a dozen times, and also contained blatantly-stolen GPL'd code that put the company in legal jeopardy. When the company hit a speed bump, B had better "productivity metrics" so A got laid off first. –  Bob Murphy Dec 15 '10 at 19:24
add comment

Some ideas:

  1. features implemented
  2. bugs fixed (also account for bugs later re-opened by QA)
  3. user complaints resolved (note that it's not the same as 2 - one serious bug may be pain in the neck while 100 typos may not be that important)

Also may want to track:

  1. Code coverage by tests
  2. Code coverage by internal documentation
  3. Feature coverage by external (user) documentation
share|improve this answer
9  
Bugs fixed is a notoriously poor measure. It actually encourages shipping of poor software giving them more bugs to fix and be rewarded for. –  Jon Hopkins Dec 15 '10 at 9:14
    
@Jon you are right, this should be correlated with the source of the bugs (thus e.g. tests). But unfortunately bugs do happen, so somebody has to fix them. Goodhart's law states that every metric will be abused anyway. –  StasM Dec 15 '10 at 9:54
1  
And some bugs are fixable in minutes, while other in weeks or month –  Xavier T. Dec 15 '10 at 10:17
2  
i18.tinypic.com/34461wi.jpg –  Wonko the Sane Dec 15 '10 at 14:20
add comment

Measure in the same way as you are measured by the customer. In terms of functional code, but on a smaller scale.

Make short goals - a week or two - and see if the programmers fulfill the goals, and do so in a satisfactory way.

I would strongly suggest peer review of code, as this allows you to catch bad code up front.

share|improve this answer
add comment

How about Rate of sales of the product/service.

Sometimes I've heard it's called a commission/percentage of the gross

People buy good products, don't they ?

The business wants to sell the product ( or maybe service, same difference for this)

So if that is what you want, measure it.

A bit like saying people who design a car that gets good reviews & sells well have done a good job really.

Now adopt this metric and programming team will want to interact with sales guy for two reasons.

  • Promsing underliverable

  • Not selling product to customers effectively

share|improve this answer
add comment

Writing code/Programming is not like putting a hammer to a nail. Much like "writing" in general, it's not something that you can apply typically metrics too - in my opinion.

Couldn't you simply look at their check-ins, or what they've done through peer-review, code-review?

Or you know, if they actually produce working code and solutions that fix problems?

share|improve this answer
add comment

Use a methodology, whereby the written documentation marries up to the code written. Start the week deciding on what needs to be done, get an agreement, then wait till the end of the week to see if its been done or not. Keep tasks small and measurable as in how many days. I don't think you need to measure the programmers work per say, but the agreement on what is to be delivered and when is a must for control.

Second part to this solution would be peer-to-peer code reviews which is backed up by some sort of Versioning system that makes who did what and when traceable. If the consensus is that the code is good then your onto a winner, if bad, then find out why and how it could be improved.

Time and motion studies are a no no as far as I am concerned, some code such as Regexes or some really hard logic can take days to develop but may only form a couple of lines of code. The only true measurement is deliverables on time, on an agreed time.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.