Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We have a large (1200+ hours) website that has a lot of technical debt. This is mainly caused by the following (usual) reasons.

  1. Multiple programmers who come and go during development.
  2. Change of specifications during development.
  3. Numerous added functionalities added (in a short time).

The customer wants alot of new functionalities, and that basically comes down to working on this project weekly for 10+ hours.

Due to the technical debt, we spend A LOT of hours fixing or investigating problems, that usually find their origin in one of the following:

  1. A shameless, silly bug that makes people cry.
  2. A new feature results in the above because we hadn't foreseen all the places the new feature would have an influence.
  3. Some other problems we have faced (f.e. server migration, upgrades)

We have issues daily and we have tried to following things to put this to an halt:

  1. Created technical documentation regarding the import, payment and general working of the website.
  2. Have meeting at the start of the week - discussing the current issues or improvements and how they should be tackled.
  3. Have a test-plan. Programmer A test B, B tests C and C tests A. Then our Project Manager will throw in some tests. Regarding the impact of the feature we throw it on a staging environment and let the customer check for itself.

The problem is that the problems keep happening...and somehow we can't get a grip on it. New features still cause bugs, and old bugs keep saying hello. Somehow - perhaps due the size of the project - we can't seem to get a grip on this project.

I assume there are alot of programmers working on larger projects then this. That is why I come to my question:

What can we do, or what do you do to avoid these problems on large projects?

Minor edit, extra info:

  1. We use version control (SVN).
  2. We have DTAP development process.
share|improve this question
2  
I'm not sure there is a specific enough question here other than, What is the right way to develop and maintain a large web application? –  JeffO May 20 '11 at 9:25
    
I tried to make it as specific as possible. I'd like to hear people's opinion on our situation and what to improve, or share their own experience and how they approached this problem. –  Wesley van Opdorp May 20 '11 at 9:34
    
Do you have a build engine? Which builds deliverables? Every time somebody checks something in? –  user1249 May 20 '11 at 10:26
    
I had to look up DTAP: phparch.com/2009/07/… –  Tangurena May 20 '11 at 13:23
2  
Too bad Kafka was too early to write about software systems. –  David Thornley May 20 '11 at 14:29
show 1 more comment

9 Answers 9

up vote 10 down vote accepted

I'll play devil's advocate, having seen far too often how this turns out: You can't cope with it. I guarantee you're the only one who actually sees a real problem with the system as it is, or else you wouldn't have to ask how to cope with it because the company culture would be one to stamp out bugs and fix the code wherever possible i.e. operating how real professionals work.

I bet it's too large to start writing unit tests, because it hasn't had anyone who knows how to unit test before you (and with luck other people on your team) and it's impossible to know where to start, and maybe even impossible to test because it relies on exact implementations and concrete data, so it would take far too long to strip all of that out to interfaces, mocks, stubs and the like to be able to test it in the first place. I also bet you can't just go and refactor what needs to be refactored because it's too tightly coupled and, since there are no tests, who knows what will be broken by fixing bad code. In short, it's probably become too cancerous to seriously fix, but of course it can't just be cut out and start fresh.

You're fighting a losing battle, my friend. Either you'll burn out from frustration and eventually quit or go insane, or if you complain about it long enough trying to get others to realize the issues, they'll think the only issue is you and you'll be shown the door.

share|improve this answer
1  
+1 for prescience. I feel like you followed me around at my last place of employment and started taking notes. I do want to comment and say that it doesn't have to be as dire as you described it. I have SEEN real change happen with bad management when a motivated Type A personality who understands the problems can become engrained in the office politics enough to become effective. A lot of times it is like steering a BIG boat, it takes a LONG time for the massive clunker to make a 180 degree turn. –  maple_shaft May 20 '11 at 12:30
1  
Sadly, what I described has basically been the story of my development career. I can't play office politics so the people and the people who do aren't "Type A" personalities at all (or they are but don't understand the problems) so nothing gets changed except, usually, me. –  Wayne M May 20 '11 at 12:38
1  
Hang in there. I am not going to say it gets better, just that it CAN get better. Most of my career was toxic environments like this. Probably half of software development shops have this problem to some degree, it just seems like it is more prevalent than it really is because these places are ALWAYS HIRING, turnover tends to be bad. Assuming pay and benefits are comparable, people tend not to leave a shop that utilizes industry standard best practices. I got better at spotting these dysfunctional work environments on interviews, trust your intuition, it will nag you if it senses wrong. –  maple_shaft May 20 '11 at 12:46
1  
cont... Listen for key phrases like, "We are moving towards Agile" for instance, a sign that development is pushing for it but the culture rejects it. Ask what happened to your predecessor or the person you are replacing, how long has he been on that project or with the company and ask about the team and how long they have been with the company. If the interviewer shows any hesitation about divulging this information then that is a red flag. Check out glassdoor.com, do some research on the company before you accept an offer. I work in a great job now, that didn't happen by accident. –  maple_shaft May 20 '11 at 12:54
    
Looks like my pessimistic view didn't sit well with someone.. anyone care to explain the downvote? –  Wayne M May 20 '11 at 15:07
show 1 more comment

Unit-testing things is a good starting point if you're not doing any. At the very least they'll protect you from adding new bugs when fixing old bugs.

Source control also helps, unless you're not using it. The blame and log features, in particular, are wonderful to nail down how/why a buggy piece of code ever got committed.

On the customer's end, I've found that discussing price and (lengthy) delays as soon as changes/additional features are requested works reasonably well, as does charging for the time you spend discussing/designing them. Frequently, customers will decide that on second thought they can wait.

(By contrast, if you immediately dig into specs and implementation ideas with him, they'll typically set you up for a "oh, I thought we had agreed you'd do this anyway" or (worse, after several days of back and forth on the specifics) "but look, it's designed already and we what we discussed doesn't sound that hard!".)

Last but not least, I've found that being upfront that I only read emails once per day (upon arriving at work), and that I've a phone for anything more urgent, leads to a tremendous productivity increase.

share|improve this answer
add comment

I suggest you add some CI-based testing, primarly on the areas that break most frequently. That will help you increase quality as work is being done on the project.

It's also becomes more apparent which areas/functionality break more often and thus it's easier to decide which parts need refactoring, or at least increased testing.

Adding more manual testing risks having the project go the wrong way in terms of $$$ & time required per feature added.

Some code review is good to, but maybe that's part of the A->B->C->A testing scheme. (Maybe code review in the other direction?)

share|improve this answer
add comment

Let me throw a fable at you. You were taking a walk with a person earlier in the day down the street and you reach your destination. The person you are walking with quickly finds out that he lost his ring somewhere along the way so you both decide to backtrack and go searching for it. The person you are walking with quickly stops at a lamp post and begins looking frantically. You say, "Why are you looking there at the lamp post when I think you might have lost it when we cut through the alley?". He replies, "I know but the light is better here."

I have been in this situation more than a few times and I have noticed some commonalities. These kinds of maintenance nightmare projects are typically run in a process heavy environment with heavy oversight and process improvements imposed by management. I am not saying process improvements are a bad thing but more often than not the types of process improvements that Management will typically want to enact have two key points.

1) They generally don't disrupt the office politics and balance of power. 2) They are successful at creating the illusion of control by management rather than strike at the heart of the issue.

The "light is better here" management think typically goes about by saying, "Every new feature must have a detailed tech spec", or "Lets have an hourly status meeting everyday to discuss issues and how to overcome them."

Neither of these things really strike at the heart of the issues and they might just decrease productivity but they certainly validate the illusion of control by management.

The only true changes that you can help push for would be ones that shake things up. I suspect though that your monstrosity of a web site is probably beyond repair at this point and you would be further ahead to re-architect and rewrite. For the future however you can keep in mind the importance of Agile methodology, Continuous Integration, Test-Driven Development, Code-Reviews, and business requirement specifications that are regulated under strict Change Control procedures to help minimize scope creep without schedule adjustments.

These kinds of changes truly require a change in the way of thinking at the management level and in my entire professional experience I have never encountered this to happen without some kind of middle management level shakeup. I hope that isn't too discouraging as you should try for what is right regardless if you are fighting an uphill battle, because you will likely encounter fierce resistance by people who love the status-quo.

share|improve this answer
add comment

I've been in the same spot some time ago. I'm not anymore thanks to two simple rules:

  • Every week one, or two days are spent fixing/rewriting most hairy parts of the app. No bug hunting, no new feature development.
  • While implementing new features we strive to get it right even when we spent more time than customer is expecting.

The only problem is to get other people to respect them. The easy part surprisingly was customer. Can't really explain why, but somehow we have convinced him, that when we work on a feature a bit longer it's better for everybody. Respecting the first rule turns out to be more problematic, but also we feel it helps us a lot. It guarantees steady progress as different parts of application are getting better.

share|improve this answer
1  
+1, but this is often the single hardest thing to get since usually the "customer" doesn't care about quality and sees fixing hairy parts of the application as time that could be better spent designing new features. I wish I could do something like this at my job, but whenever I bring it up it's "No, they want to see new features added, not fixing stuff that works" –  Wayne M May 20 '11 at 13:58
    
@WayneM Yes to this day I'm amazed that this actually works, given the attitude of some people. This must be because management run out of ideas how to reduce "the bug count" and decided to try our approach. –  Jacek Prucia May 20 '11 at 14:33
add comment

Code reviews. Unit tests. Real QA Testing. Specification collection process and incremental development - these are some things that should resolve most of your issues.

Also don't let the customers directly ping your developers - This is usually the most unproductive way of solving issues. Hire a good Program manager who will form the interface between your customers and developers. His job would be to know the product end to end, current status, future directions and so on. Any time the customer wants another new feature, he should be able to give the current list of items and show the customers what will get bumped off if this new request is to be taken.

Process is bad when it is used too little or too much. I think you are at the former state.

share|improve this answer
add comment

As Deni mentions, if you are able to add unit testing to the project, then this would help you. Have a test which covers a part of the system your about to change/fix, and so when your refactoring code, use that test as a guide to make sure your not breaking anything.

Also, triage the most broken parts of the code. Try to put those worst affected into a list of risks and manage those risks independently. Try to get an idea of how much broken code there is in the codebase by querying where the bugs happen most. You can then list the affected area by the bug count (or issues reported, whatever works for you.).

Patching up and cleaning the code will take time, but if each dev on the team can leave the code a little bit cleaner then it was before they toched it, then over time the codebase will improve. If your looking for a quick, army style, get it sorted now solution, I'm doubt that there is anything practical (or recommended) which would help.

Cheers. Jas.

share|improve this answer
add comment

Write clear functional specifications; pedantically so if you can bear it and review functionality against those specs regularly. The less of an idea a dev has about what he is supposed to be developing, the less chance there is of it being the way it is supposed to develop.

Before you start writing code do some up front design work; this doesn't need to be perfect, or huge, or contain UML, but it should outline a fairly solid solution to the problem that needs to be solved. As far as I can tell the less software is planned, the worse it is. Discuss the design before you start working on it.

When you start working on an area of the code that is clearly bad and really hindering your progress; stop adding to it, step back from the problem, work out how you could redesign the architecture so that the hindrances weren't there and so that it would be more adaptable in the future. The longer you leave it before you address technical debt the harder it will be to address it without a full rewrite. I'd say it's an exponential thing.

Design tests that test behaviour and don't couple tightly to your architecture. It isn't very fashionable, but I would say don't start testing until the true purpose of your code is clear. Specifically don't start testing until you know what you really want to test; IMO a poorly thought out test is worse than no test. And the more tests you have the harder it is to change your code internally. Treat your test code as you would production code; it needs to be planned and well written.

Do regular / daily code reviews: this is more about sanity checking to make sure the dev hasn't gone to far off course. Use these sessions to plan out the next days work. There may be days when these take 5 minutes, or 1 hour; the point is to keep a dialogue open and give developers a chance to discuss their work with other developers and seek advice. Do some pairing sessions on tough parts of the code, or to prototype ideas, but let people have their own time to work.

Make it easy to build and deploy your code. Try to keep build times short. The easier it is to build the more it will be built, the quicker it is the more it will be built.

Adopt coding standards and enforce them rigidly. This should cover everything from where a project should live in the file system to the casing of a private const. This might seem pointless and annoying, but good habits are the cornerstone of a development process.

Fundamentally I don't think that the process you use matters so much, fashions come and go. What really matters is that you are professional about how you develop software and that you are disciplined in your practice.

share|improve this answer
1  
-1: Write clear functional specifications; pedantically - I strongly disagree because time and energy spent writing "pedantic, functional specifications" (that will quickly become obsolete) is time and energy that you can't spend writing functional unit tests that validate the code every automated build cycle. –  Jim G. May 20 '11 at 13:30
1  
"that will quickly become obsolete" is the biggest fallacy in the whole of software management. If they become obsolete then update the FS so that they aren't. If you don't have a proper FS how on earth do you know what tests to write or if your software actually does what it want. For me this is everything (and there is a lot) wrong with agile: let's just start writing code, the tests are everything. Documentation is a waste of time, getting things clear and explicit is a waste of time... –  user23157 May 20 '11 at 13:58
1  
You both make valid points. Strong functional requirements are necessary for a solid testing practices, however when the project is already being mismanaged then this will help very little. –  maple_shaft May 20 '11 at 14:18
2  
I take your point, but in my experience not knowing what is being developed is the seed of mismanagement. –  user23157 May 20 '11 at 14:49
    
@B Tyler: ...In my experience not knowing what is being developed is the seed of mismanagement. - 100% agree. We just disagree on the remedy. –  Jim G. May 20 '11 at 15:41
show 2 more comments

I'd start off by designing and automating smoke tests, and throwing them into CI environment. Those should be functional. When the customer tells you that something should work so-and-so, ask to write it down, so that you may refer to it later. When you see a certain solution in the software, ask questions, and as soon, as you receive answers, incorporate them into knowledge base, and make them traceable.

Assure, that the basic functionality for positive cases works. Then start building incrementally tests for incorrect data handling, placing defects where deemed necessary. Have a long and deep discussion about priorities, and have the test manager aware thereof, so he can assign testing time accordingly. Don't attempt to automate everything, but as soon, as some testcases make sense to being automated - don't hesitate.

In general, use testing for increasing trust in the product, and not as the tool that will instantly increase quality. Be calm, yet assertive :). Maybe attempt going agile, but only if you absolutely, positively may engage a certified PM. Introducing Agile by a person who don't know Agile, will most probably kill the project.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.