Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Some Background:

Our company is growing very quickly - in 3 years we've tripled in size and there are no signs of stopping any time soon. Our marketing department has expanded and our IT requirements have as well. When I first arrived everything was managed in Dreamweaver and Excel spreadsheets and we've worked hard to implement bug tracking, version control, continuous integration, and multi-stage deployment.

It's been a long hard road, and now we need to get more organized.

The Problem at Hand:

Management would like to track, per-developer, who is generating the most issues at the QA stage (post unit testing, regression, and post-production issues specifically). This becomes a fine balance because many issues can't be reported granularly (e.g. per-url or per-"page") but yet that's how Management would like reporting to be broken down.

Further, severity has to be taken into account. We have drafted standards for each of these areas specific to our environment. Developers don't want to be nicked for 100+ instances of an issue if it was a problem with an include or inheritance... I had a suggestion to "score" bugs based on severity... but nobody likes that. We can't enter issues for every individual module affected by a global issue.

[UPDATED] The Actual Questions:

How do medium sized businesses and code shops handle bug tracking, reporting, and providing useful metrics to management? What kinds of KPIs are better metrics for employee performance? What is the most common way to provide per-developer reporting as far as time-to-close, reopens, etc.? Do large enterprises ignore the efforts of the individuals and rather focus on the team?

Some other questions:

  • Is this too granular of reporting?
  • Is this considered 'blame culture'?
  • If you were the developer working in this environment, what would you define as a measureable goal for this year to track your progress, with the reward of achieving the goal a bonus?
share|improve this question

migrated from superuser.com Mar 8 '11 at 19:04

This question came from our site for computer enthusiasts and power users.

10  
Blame culture = bad. –  NickC Mar 8 '11 at 19:10
    
@renesis then suggest alternatives? –  philwinkle Mar 8 '11 at 19:17
    
I wonder why this didn't get migrated to the project management SE –  Doug T. Mar 8 '11 at 19:19
    
@Phillip It's too long for comments, but the reason I didn't post an answer is that your question was how to accomplish it, not whether or not to do it. But there isn't any shortage of conversation on the topic if you Google it. –  NickC Mar 8 '11 at 19:35
2  
I wouldn't work in a company where you get measured strongly on the bugs you create. That's how people will stay away from hard problems, that's how teammates will turn on each other. You can foster quality and responsibility with less reactive methods. I would start by trying to get rid of the idea of using that metric. You can still establish goals of completion and quality on a team scale, teams will self-regulate if you allow them (and if you hired right). –  ale May 20 '11 at 12:31
show 5 more comments

6 Answers

up vote 14 down vote accepted

I would strongly suggest that management reconsider trying to track things in this level of detail. It's going to be inherently subject to gaming.

I've seen clients attempt to do something similar but at a group level rather than at an individual level. What inevitably happened was that there was a strong incentive for each manager to get their group's bugs classified as low priority or classified as an enhancement and people started to get very defensive whenever there was a suggestion that there was a bug in their code. From a metric standpoint, it looked like code quality was up tremendously month over month but that was only because anything that didn't cause a total systems failure was being tagged as an enhancement or a low priority bug.

What is management trying to achieve by tracking developer effectiveness? If they want to improve overall code quality, it probably makes sense to have a feedback loop from the bug tracking system that tries to determine why a bug made it to post-production and what should be done in the future to prevent similar bugs. It may be that the requirements were unclear or inconsistent. It may be that the developer was sloppy. It may be that the QA department needs to more thoroughly test certain data conditions. It may be that management made a calculated decision to rush some functionality to hit an external deadline.

But if the intention is to improve code quality, this feedback loop has to be reasonably safe. That is, people have to have reason to trust that admitting to reasonable mistakes isn't going to cause problems for them down the line at review time. For example, if the QA department missed a bug because they're supposed to do dozens of poorly documented manual steps to test something and someone innocently missed a step, they have to feel safe in admitting the mistake so that management can identify the fact that they need to allocate time for someone to automate more of the QA process. If the problem is that the project manager made a last-minute change to the requirements which caused the developer to rush a change in and for QA to skimp on the testing, everyone needs to feel safe enough to discuss how they might have handled that situation differently in the future. If the folks that are most willing to admit to making mistakes in order to improve the process are the ones that are getting lower ratings during reviews because everyone else is pointing fingers and denying responsibility, you're not going to have a positive effect on code quality.

If you are going to report some sort of numeric KPI, the most meaningful numbers will have to come from something that the development staff cannot reasonably game and will have to come from a very coarse level of granularity. The set of numeric indicators that the development team cannot game tends to be very application- and organization-dependent. For example, you may be able to drive some metrics by parsing the application logs to look for certain types of errors (i.e. how many times did a user go to an error page because of an internal error). You may be able to drive metrics based on things like how quickly the software allowed a user to accomplish a particular task.

The set of things that the development team cannot game, however, is likely to result in metrics that apply to large swaths of the development organization. Performance-based metrics (i.e. our logistics software has improved inventory turn times 10% this year) are going to require that the entire team is working together from the developers to the DBAs to the hardware group. So they're not going to be meaningful to track how productive an individual or even a group is. But they are going to be the sorts of metrics that you actually want senior management to manage to. Senior management shouldn't care whether Jimmy the Developer is buggy code (though Jimmy's immediate manager should be aware). But they should be aware if Jimmy's buggy code is causing the call center's customer lookup operation to waste 10 hours of call center rep time every day or if some cool-looking new feature is chewing up 50% of the available CPU and slowing the rest of the system down.

Lower level managers can participate in the QA feedback loop and will interact regularly with the various development teams. It should be clear to them which individual developers are particularly strong and which are particularly weak. It should be clear where the recurring pain points are whether those pain points are communication or politics or developer strength. Having numeric KPIs at these low levels is going to be exceptionally difficult-- they are going to be too easy to game and they are going to create some perverse incentives. A developer's manager should understand whether a developer that is being assigned a lot of bugs is a weak developer that needs mentoring or a strong developer that is being exceptionally productive or an unlucky developer that has responsibility for a legacy module that is known to be exceptionally complex or buggy.

share|improve this answer
2  
+1: Metrics (i.e., numerosity) => gaming the system => perverse incentive => undesirable consequences. –  S.Lott Mar 8 '11 at 19:35
    
I have updated with more specific questions. I hear where you're coming from here - we're doing everything we can to encourage clean code and minimization of errors from the actual developer perspective - the idea is to somehow effectively communicate this to the management who wants to see measurable numbers. –  philwinkle Mar 8 '11 at 21:05
    
@Phillip - Updated my answer –  Justin Cave Mar 8 '11 at 22:44
    
+1: everyone needs to feel safe enough to discuss. Metrics don't create this feeling. –  S.Lott May 20 '11 at 11:55
add comment

How do they integrate source control, bug tracking, developer 'effectiveness' reporting, and other managerial aspects of the business?

They don't.

Source control is a good choice of tools and a rigid policy that the only code which exists is under source control. Everything else will not be trusted.

Bug tracking is a good choice of tools and a rigid policy that unreported bugs don't exist.

There's no real need for "integration". Some folks like to try and put bug numbers on code checkins. However, when one checkin fixes multiple bugs, this seemingly simple thing can start to get complex and messy. Too much time spent fooling with the tools is time wasted.

The Agile Manifesto is important reading. http://agilemanifesto.org/

Developer 'effectiveness' is almost entirely non-technical and simply requires real managers really manage real people. There are no tools. Unless you consider a conversation to be a tool.

Counting issues and their severity is a big waste of everyone's time. It's easy to tell who's causing problems. Managers need to simply sit with people and talk to them. A few minutes a day will do amazing things to enhance the manager's understanding.

Also, standard practice outside IT is to be sure that a manager can actually do everyone's job. Code a little. Test a little. Migrate to production. etc.

"other managerial aspects"? What? Planning? Scheduling? Budgeting? Hiring? Promotion? Termination? Status Reporting? There are numerous aspects. They're all handled through hard work, patience and a sense of perspective.


Update.

What kinds of KPIs are better metrics for employee performance?

None. There is no measure of "performance" in a broad, vague, useless form like that.

Some specific things can be measured. But broad, vague "performance" is meaningless.

You could measure things like lines of code per hour, bugs closed per week, things like that. It works out very, very badly in the long run, but you can measure specific things. You can't measure "performance" until you define it.

What is the most common way to provide per-developer reporting as far as time-to-close, reopens, etc.?

"time-to-close"? Bug tracking? Sure - bug tracking systems have time stamps. It's trivial to measure this.

It doesn't mean anything and has unintended consequences that are very, very bad.

When you measure time to close the very first thing that happens is the time-to-close drops from days to minutes.

And the number of unsolved, but closed tickets, goes right through the roof.

Why? A new ticket restarts the clock on time-to-close.

So. If you can't solve it right now, you close it. Then you work on it. When the user reopens it, and you can solve it, you solve it and close it.

Everyone does this when you measure time-to-close. The time-to-close number means nothing.

Do large enterprises ignore the efforts of the individuals and rather focus on the team?

Actually, yes.

Some other questions:

Is this too granular of reporting?

Yes.

Is this considered 'blame culture'?

Yes.

If you were the developer working in this environment, what would you define as a measureable goal for this year to track your progress, with the reward of achieving the goal a bonus?

Developers measurable goals have to do with features created for users from development backlogs.

share|improve this answer
    
See my update above - I have asked some more pointed questions. The idea is to provide metrics not to 'blame' by but more to set goals by. –  philwinkle Mar 8 '11 at 21:06
    
+1. Esp. for "real managers really manage real people." –  Macke May 20 '11 at 7:07
add comment

I've worked in environments where they tried to use blind metrics to assess individual developers and I can tell you that I haven't seen a more toxic, demotivating environment in my life. Everyone who was at least slightly better than awful jumped ship at the first opportunity. All those who stayed behind were either professional work-dodgers or sat all day paralysed with fear. Because if you don't do anything, you can't make a mistake.

What you need is a manager who knows a thing or two about development and knows who's good and who isn't. There's no substitute for human judgment and communication. Metrics are fine and they should be used to control the quality of the software but not the quality of developers.

If all they want to see is numbers, then I'm afraid they're bad managers. You don't need managers to compare numbers, even a well-trained monkey can do that. But if push comes to shove, you should measure the average time it takes for someone to fix bugs, not the number of bugs they "create".

Update: Although hilarious at first, there's a serious lesson in there.

share|improve this answer
add comment

I understood your problem, but I would like to suggest that a manager get involved in the activity. Data solely can't help to solve the problem.

Before solving any problem, we should take some time to grow a strong team work in a friendly way. There should be no gap inside the team.

Before performing any work, one meeting should be organized in the morning & every activity should be detailed in the notice board by the responsible member. And at the evening meeting, morning activities shall be revised. That will help to reduce the failures and solve the problems on time.

Root cause analysis is the important prerequisite of any activity.

Team training is the most important task of any project. Only good team (well managed, well defined, well disciplined, positive thinkers, well qualified, friendly performers) can solve every problem and perform every task.

Two things are important for a good team :

  • Motivation
  • Organized behavior, which is the responsibility of the owner of the firm.
share|improve this answer
add comment

If managers don't want to directly come in and evaluate each design/bug go for peer reviews. Also make it clear to the team that there is a performance curve that has to be fit (10% exceeds. 10% low perf and 80% medium) - to avoid everyone giving everyone else a good rating. Ask for SMART assessments and situational examples to backup each claim.

This coupled with basic bug tracking/system health analysis should give them a very good overview about how well the teem is doing.

share|improve this answer
add comment

The short answer is that developing software is a team effort. It makes little sense to try to track individual developers by metrics because an individual's metrics values can be skewed and biased by many factors not in that person's control. For instance, some bugs take longer to duplicate and solve than others. A developer that is familiar with a section of code may be able to fix and test a problem in a matter of hours where another developer who is not will take longer. The root cause of a problem may not be in the developer's assigned responsibilities, and it may take a while for that developer to characterize the issue and hand it off to someone who can actually fix the problem. It makes more sense to track the team as a whole.

Here are answers to your specific questions:

How do medium sized businesses and code shops handle bug tracking, reporting, and providing useful metrics to management?

There are many bug tracking tools around. Bugzilla is free and works for tracking bugs. Many open source efforts use Bugzilla. It is also used in industry. There are commercial bug tracking systems available as well. Some commercial tracking tools integrate nicely into test tracking tools and source code control, and even make attempts are integrating into requirements. I will not name names here as I am bound to miss some. Wikipedia has a pages comparing bug tracking software, and comparing project management systems. In addition to tracking issues, a team (usually composed of at least the technical lead, the test lead, the project manager, and a customer (or customer advocate) is formed to periodically review issues. This team reviews, assess, prioritizes, assigns and otherwise dispositions the new and open issues. The meeting period can be as frequently as daily (during some phases of the testing) or infrequently (weekly or bi-weekly) during early development stages.

What kinds of KPIs are better metrics for employee performance?

I have never observed any QA metric that has been successfully applied to individuals. I have never seen. The only individual metric I have seen applied to developers is Performance Against Schedule (but only when the developers set the schedule themselves). In rare cases, I have seen developers removed from teams if they are consistently not fixing assigned issues in a timely manner. These cases were obvious to all team members.

What is the most common way to provide per-developer reporting as far as time-to-close, reopens, etc.?

The main reason individual developers are tracked is to monitor their workload. Someone with many current open issues is not necessarily a good candidate to assign additional issues to. After the fact, during the project post-mortem, QA metrics can be used for process improvement, architectural improvements and other strategic planning. For instance, a component with a statistically significant number of reopens should be looked at more closely. This indicates that it may be tightly coupled, or poorly designed.

Do large enterprises ignore the efforts of the individuals and rather focus on the team?

I have worked in several different software development organizations in my career. In all cases, QA metrics were used to monitor the health of the software being developed, not to monitor any specific individuals. On the other hand, individual performance appraisals were based on (perceived) performance. During the development phases, this usually meant good performance to schedule on critical sections of the requirements, design and code. During integration and test phases, this usually meant rapid response to open issues. Technical leadership is supposed to keep these factors honest.

Note: I assume from the way your questions are phrased that you are working in an environment that is not fully participating in continuous integration. If it is practical to do so, push the organization in that direction.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.