Methodologies of a high class development team

Question

I've recently been tasked with increasing standards and efficiency within a development team. I've read similar posts about the makings of a good development team but these often discuss the human elements such as "passion for learning", "being a team player" etc. These are all of course important but I'm particularly interested in ideas surrounding practical methodologies.

Some are just plain obvious, for example, my first implementations are:

• Implementation of Source Control - well, do I have to even explain why?
• A structured method of deploying builds that everybody adheres to - makes deployment easier for all involved
• Coding standards - ensures everybody is working in the same way and (hopefully) reduces technical debt
• Encouragement to make use of the developer community e.g. StackOverFlow and the like - keeps us in the loop with the big bad world and brings in ideas from outside our office walls

What practical methodology does your team employ that you feel makes a tremendous difference to the productivity and overall "goodness" of a development team?

Answer 1

Here are some things you can do:

1. Implement a source control system.

Reason: To keep a history of code, and also to enable branches to be created for new developments, which in turn means it's possible to release some code to testers, carry on with development on a new branch, then fix bugs on the release branch if the testers find bugs - this is the best route to creating stable releases.

2. Write unit tests

Reason: If you don't unit test your code, how do you know it works (in all scenarios)? How do you know it still works when somebody changes something? You should make sure unit tests are just that - unit tests - and not integration tests. That means that if you have a data access layer, you should test it without it needing to connect to a database (or if you really must, also have scripts to do setup / teardown of data). If you have code which makes calls to services, create mock service endpoints in code (so you don't need to actually connect to the service. Of course, you need other unit tests to test the service code in isolation). Additionally, every bug that gets raised should have a unit test which replicates the bug, to help ensure no regressions occur.

3. Implement a continous integeration system

Reason: Continuous integration means (essentially) that when code is checked in to the source control system, a build server then attempts to rebuild the system (and run the tests), and will report success or failure. In .NET (for example), you can use Nant + Ivy + CruiseControl (all free) which means a developer can do a full build on their machine (before checking in) which gives confidence that their code wont break anything, they can then check their code in which triggers a build on the build server. Other developers then have confidence that if the code builds (and all tests pass) on the build server, then getting the latest version of code from source control will also work on their machine. This encourages developers to check in early and often. It also makes it possible to create automated releases and branching from the build server.

4. Implement coding standards

Reason: Badly formatted code takes longer for a developer to read and understand. It also shows a level of laziness from programmers if they can't be bothered to format their code correctly or consistenly with other members of the development team. In .NET you can use Stylecop to enforce coding standards.

5. Peer review code

Reason: Programmers should not be embarrassed by their code, so they should have no problem showing it to other people. Every task (story / work item / issue / whatever) should have an associated peer review. The peer review should be quick and simple (eg. (1) is the code unit tested, and what is the percentage coverage, (2) does the code follow coding standards, (3) does the code match the design (if there is one) -or- is the code well structured / could it be refactored)

6. Track everything

Reason: If a bug isn't tracked, how do you know how many bugs you've got? If tasks aren't tracked, how do you know what's been released, and what's left to do? So get a good issue tracking system, and put eveything in there (tasks & bugs). When code is checked in to source control, the developer should note (in their check-in comment at the very least) the task or bug number that the code relates to. You should also keep a record of the peer reviews that have been done, and what version of code everything relates to (eg. what version of code does a bug affect, what version of code was peer reviewed, and by whom?)

7. Automate everything

Reason: It's much quicker, and less error prone, if everything that can be automated, is. So braching code should be automated (using the continuous integration system), releases should be automated, and you can write automated acceptance tests. If your testing is not automated in any way, each new release will take longer and longer to adequately test, due to the volume of regression testing that has been built up. And the testers should just be testing the new stuff, not testing the release process, or testing things they tested in the last release.

Answer 2

One verb: care. If everyone on the team cares - about each other, about the customer/client/enduser, about the people whose hands are on the keyboard and mouse when the software is live, about the people who sign the cheques or hand over their credit cards, about the profitability of their employer - then almost any methodology will "work" and the team will accept the methodologies you suggest and make those methodologies successful.

People who don't care follow methodologies in a very passive-aggressive way and frustrate people who are trying to make things better. People who don't care need 50 page instruction guides on the optimum size of a checkin/commit or 20 page documents on how to comment your code. People who care do the right thing. When they see they did something not quite right, they do it better next time.

Therefore you want to avoid creating a methodology that frees people from caring about each other or their users. You want to surface as much information as you can about the consequences of the choices we all make every day. And you want the company/organization to show that it cares about the developers, too. Automated builds, automated testing, and continuous integration say "we don't want to waste your time." Source control says "we don't want to lose your good work, or to have it lost by someone overwriting it accidentally." And so on.

Take each proposed part of your methodology, be it process or tool, and ask if it helps developers care about each other, the company, the users, or if it gives them an excuse to hide behind. Ask if it sends a message that the company values and cares about the developers, or that it doesn't trust them to do the right thing. If a given tool or process could be interpreted either way, make sure you communicate to everyone the "we care here" interpretation.

Answer 3

[Disclaimer: the opinions below are my own, they might get on your nerves, but I do try to back them up with facts]

Note: It ended up being much longer than anticipated, so here is a summary :p

1. Use a Distributed Version Control System
2. Implement Automated Testing
3. Automate Automated Testing
4. Test Deeper
5. Test Review
6. Code Review
7. Design Review
8. Coding Standards
9. Consistent Formatting
10. 3rd Party Code ?
11. Task Tracking
12. Automate Delivery
13. Document

1. Use a Distributed Version Control System

Among good practices, you'll find the use of a VCS. Unfortunately it's insufficient. Trash CVS or SVN and get a modern Distributed VCS (preferably Git, or Mercurial otherwise).

DVCS have several advantages over traditional sever-centric VCS:

• clones instead of check-out: makes branching easy, makes sharing code easy, makes working offline possible, ...
• improved merge strategies
• no longer necessary to back-up the server since everyone get a copy (which means you cannot screw-up a restoration...)

2. Implement Automated Testing

Testing the same things over and over is really boring. Yet you cannot have any confidence as to the quality of your software without testing it!

That is why automated testing is a must-have, since it relieves the burden of re-testing while keeping its advantages.

3. Automate Automated Testing

Even with automated testing, you still need to run the tests. It takes time. I gets in the way. "And I haven't changed much anyway"

• Commits to the central repository should only be possible after passing a minimal set of tests (the full test suite might be too long), this means that you do not commit, you enqueue a commit request that a bot (Hudson, Buildbot) will eventually process for you.
• Nightly non-regressions tests should take over to run the full test-suite, it's everyone's top priority to get the nightly back up to 100% if they ever fail

4. Test Deeper

Functional testing is indispensable, but it's covered by the first point.

You can go deeper though.

• Statical Analysis software, aka Automated Bug Finding
• Runtime Analysis, aka Automated Bug Finding 2

Run analysis as part of the nightly non-regressions tests. Unfortunately it's difficult to be precise here because I only know some for C++ :/

5. Test Review

Each specification change and each bug resolution should be accompanied by a set of tests. These should be reviewed to try and avoid "forgetting" something. New code should not make it into the repository without being accompagnied by its set of tests.

6. Code Review

Each new feature or bug resolution should be review by at least one person. In my team we try to involve 2 persons: one that is knowledgeable in the area (to "validate") and one that is not (to get experience).

The code review should be technical and functional, but it's too late to question design strategies (unless it's a glaring mistake).

It's the moment to enforce Coding Standards / Naming Conventions.

7. Design Review

As mentionned, it's too late when at the code review stage to question design decisions. It's also difficult to design on one's own, because design requires insight, foresight, and so on.

Design decisions should be discussed, I once again recommend a 3 persons settings:

• the proposer: responsible for prototyping / checking the documentation etc... and coming up with a design
• the reviewer: someone with experience, with knowledge of the application, often times the team leader / project leader
• the student: someone either inexperienced in the craft or in the application

It's not a good idea to let inexperienced / less knowledgeable take design decisions unguided: it's so much easier to change a paper draft than a hundred source files.

8. Coding Standards

You need one, but you do not need any one: you need a good one.

There are many dubious advices laying around on the web: don't incorporate them blindly.

A good Coding Standard is:

• technical: that's how you get objectiveness
• light: no one can memorize hundreds of points

Taking the points backward: light ensures that you Don't sweat the small stuff (C++ Coding Standards, Herb Sutter and Andrei Alexandrescu).

technical ensures that it's objective, which gives it much more change to get accepted.

There probably isn't a Coding Standard adapted to your team, build you own by taking reknowed standards and adapting them to your specific projects. Discuss each point and make sure everyone agree on each of them, you need every team member to commit themselves on this.

9. Consistent formatting

Naming Convention, Indentation Style, Bracket Style, Tab vs Space... there are countless debates in the programming community on these subjective subjects.

One thing is sure though, you need a consistent formatting style in your code base, and preferably one that everyone is comfortable working with.

This calls for setting those things once and for all. There is no silver-style, there are dubious ones, but what really matters is consistency of the code and commitment of the programmers.

Tools can assist in this task.

10. 3rd Party Code ?

The Not Invented Here syndrom is bad, but incorporating blindly thousands of libraries is not much better.

• The decision of bringing in a library or re-developping something in-house is a Design decision, and thus should involve several people (comprising the project leader)
• Restrict yourself to open-source well-tested libraries if possible (not necessarily free mind), it makes it easier to investigate bugs if you can step into the libraries calls
• Encapsulate the libraries, you don't want to have an interface change ripple through your code base like a typhoon in an atoll

11. Task Tracking

I know bug tracking is all the rage, but there is much more than bugs to be tracked, features changes should be tracked too, and planified, and accounted for.

If you have an in-house system, well, use it.

If not, there are several systems existing out there, some free, some not. Examine them, pick one. You don't want to be recoding them I think, it would be a diversion from the real goal of your team :)

12. Automate Delivery

Even the latest addition to the team should be able to deliver the software into production.

It means that the process should be documented, streamlined, and automated as much as possible.

Delivering the software should be as simple as pushing a button. Bonus points if this actually checks the delivered archive as well ;)

Oh... and obviously, reverting the deployment should be possible and automated too... we've been extra careful but Murphy is always on the prowl :/

13. Document

I am zero for overzealous documentation... but:

• It's easy to get Doxygen / Javadoc / whatever running (even if the code doesn't actually use their specific comment formatting, it still brings value)
• You need a central place to share procedures, archive meetings minutes and design decisions (with their rationale), etc...

I recommend a simple web server, preferably with a search engine. A wiki can be great. Anything easy to administrate / use, preferably light, once again it's not your primary task to administrate it, so it should not cost much.

Answer 4

XP will address most of your concerns. It is made for small teams and brings a lot of cool techiniques that can help your team, such as TDD, continous integration and coding standards. You could, for example, set up a Version Control System and a build server that continuously watches the VCS for changes, runs all the unit tests and generates an output (an exe, a jar, whatever). There are tools like Hudson for helping with that. But that is only a small part. I recomend you to read about XP. You don't need to do everything. Just go for the things your team feel confortable. You can improve your process over the time.

From my experience, another nice thing to enhance the "goodness" is to buy books. The company I work for buys any book that we want. It's nice to be able to catch one and take it home to read and learn new things.

Answer 5

Implementing a bug tracking system greatly improves team communication and reduced the number of issues in a deployed product. All the problems you come across need to be documented and "in your face".

Answer 6

I'm taking some ideas out of specific methodologies because they're good in general:

1) Every time you touch code, try to improve it - This will hopefully reduce how much bad code there is simply by people refactoring the code into a better implementation

2) Test early, test often - This way you know your code is working as you develop it. Doing this has saved me a lot of time versus projects on which I just wrote everything and tested later.

3) Don't let people break the build - Keeping people from submitting code to the main repository until its working will keep people from pulling in a broken build or accidentally promoting incomplete systems to production which is very bad.

4) As others have mentioned, a bug tracking system. When you can track bugs, you can fix bugs.

5) Bug seeding to find out approximate how many bugs you have - This works by having one group of people intentionally introduce bugs into the system and having another group check the system for bugs. Then you calculate what percentage of the bugs were the ones placed into the system intentionally. From there you can approximate how many bugs are in your system by dividing the unseeded bugs by the percentage of the bugs found being seeded. - Just saw the comments about the size of the team. For a team as big as it is currently, this is a bit of overkill. I would wait a while before doing this one.

6) Some form of workflow management system. - What I mean is something to track what is currently being worked on and by whom and how long it is taking. My experience with such systems is mainly with Kanban which is used to track and limit how many items are being worked on at a given time.

7) Use frameworks and libraries as much as possible. - This will reduce how much you are rolling your own to a minimal amount. I'm sure we've read a lot of stories from The Daily WTF which demonstrate how rolling your own can cause ... problems.

8) Don't add a lot of process in the middle of a project. - This might seem counterintuitive but as McConnell showed in his book Rapid Development, adding process in the middle of the project can result in increased thrashing. The problem is that when process is introduced it temporarily reduces productivity as it gets set up. Seeing the reduction in productivity, management adds more process to better correct the situation. On top of that, as process is added developers initially thrash more. Eventually, this self feeding loop of non-productivity results in no progress being made. Point is, don't panic if process doesn't instantly increase productivity and add more as a result. That is a dark and dangerous path.

This is an incomplete list but this is what I have off the top of my head.

Answer 7

Aside from things already mentioned. I think that having version control is not enough. How you use version control is very important. Here are some guidelines we use:

Every commit to version control should be:

1. small
2. complete and correct
3. understandable by itself
4. have one single clear objective

There are many reasons to use this practice.

Problems with large commits

Large commits are harder to review. The commit is large either because the commit contains multiple unrelated changes or the change is very complex. Break the change down into a series of smaller steps so that any future reviewer can easily follow the chain of reasoning. Occasionally a checkin may be large because it contains the result of an automated tool (e.g. renaming a widely used function), in this case you need to plan how the reviewer will confirm the change in advance.

Large commits often mean that the code is checked out on the developer's local machine and not integrated for a long time. This increases the chances of conflicting changes between developers.

Problems with incomplete or incorrect commits

Incomplete or incorrect commits break the invariants of some or all of your code. They cause code to not behave according to its contract. If you have multiple developers working on the code at the same time this could interfere with what they are doing. Some event, like a fire alarm or 5.30pm or a sick day may mean that the incomplete code gets released.

Each commit should take your codebase from a working correct state to another working correct state. You should aim to be able to build and release every revision of your project's source control tree.

Problems with incomprehensible commits

If a bug is found between two versions, or for the purposes of code review you need to examine the version control history each commit should be clearly understandable. Both the purpose (commit comment?) and the actual code change. For instance, do not paste code from some where else in the system in one commit and then remove it in a subsequent commit. The reader of the commit will not be able to tell from the first revision where the new code has come from. Move the code block in a single commit.

Writing clear tests to accompany each commit also helps here.

Problems with commits that have multiple objectives

Commits with multiple objectives are difficult to revert. In particular, if one of the objectives is wrong but the others are OK then it may take significant error-prone work to un-pick them. The same logic applies to creating patches for older versions (branches). You should be able to create a patch for an individual functionality change in isolation.

Answer 8

What do bug trackers, continuous integration and shared source control have in common? Transparency. Great teams are transparent -- any member can instantly know what is going on with the project, the build, what is broken, what is being fixed and who is fixing what.

Corallary to this is that team members know they are good and when they break a build they don't go hide in their office but say "Yup, I broked it, here is the fix" or otherwise behave like adults.