Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

It seems an infrequent but common experience that sometimes you're working on a project and suddenly something turns up unexpectedly, throws a massive spanner in the works and ramps up the complexity a whole lot.

For example, I was working on an application that talked to SOAP services on various other machines. I whipped up a prototype that worked fine, then went on to develop a regular front end and generally get everything up and running in a nice, fairly simple and easy to follow fashion. It worked great until we started testing across a wider network and suddenly pages started timing out as the latency of the connections and the time required to perform calculations on remote machines resulted in timed out requests to the soap services. It turned out that we needed to change the architecture to spin requests out onto their own threads and cache the returned data so it could be updated progressively in the background rather than performing calculations on a request by request basis.

The details of that scenario are not too important - indeed it's not a great example as it was quite forseeable and people who have written a lot of apps of this type for this type of environment might have anticipated it - except that it illustrates a way that one can start with a simple premise and model and suddenly have an escalation of complexity well into the development of the project.

What strategies do you have for dealing with these types of functional changes whose need arises - often as a result of environmental factors rather than specification change - later on in the development process or as a result of testing? How do you balance between avoiding the premature optimisation/ YAGNI/ overengineering risks of designing a solution that mitigates against possible but not necessarily probable issues as opposed to developing a simpler and easier solution that is likely to be as effective but doesn't incorporate preparedness for every possible eventuality?

Edit: Crazy Eddie's answer includes "you suck it up and find the least expensive way to implement the new complexity." That made me think of something that was implicit in the question but I didn't specifically raise.

Once you hit that bump, and you incorporate the necessary changes. Do you do the thing that will keep the project as close to schedule as possible but may affect maintainability or do you go back to your architecture and rework it on a more detailed level that may be more maintainable but will push everything back during development?

share|improve this question
add comment

7 Answers

up vote 6 down vote accepted

What comes to my mind reading this is the agile adage: tackle the riskiest and/or least well understood tasks first within the project lifecycle. I.e. try to put together a working skeleton of the project as early as possible, to prove that the concept works. This in turn also enables one to run any sort of cruel tests to detect whether the architecture really delivers its promise under real life circumstances. Also, if there is any new, unknown technology / platform / tool included in the solution, take that early on the plate as well.

If the core architecture is OK, the individual functionalities can be added and tested incrementally, and refactored when needed, with relatively less cost. Needing to change the architecture is the big risk, which one should deal with upfront. This gives rapid feedback: in the worst case, if the whole concept falls apart, we know it early and can abort the project with minimal loss.

share|improve this answer
add comment

Your example touched some of the most challenging aspects of programming, namely distributed computing and concurrent programming, which are becoming more widely used and making programmers work ever more difficult.

Even "normal" programming (single thread on one machine) is so massively complex for any non-trvial program, that it takes great skill and years worth of experience to get any good at it -- but still far away from "solved". Even on this level complexities, mostly due combinatorial explosion, far exceed capacity of human brain to fully grasp and understand. To think otherwise is foolish.

Distributed computing and concurrent programming add two more dimensions on the size of "complexity" space, which grows at least in cubic (sp?) (n^3) compared to "normal" programming. Just for example, think about some new sets of problems and fallacies we have to cope with. To even play with an idea, that you could fathom interconnections and side effects at this scale is laughable.

I clearly don't have any silver bullets, but I am quite sure the biggest mistake one can make is to think you understand it all & solved it.

Some ideas on how to cope with this all, in addition to what other answers have already covered:

  • Great humility
  • Accept that your system/program is imperfect, impermanent and incomplete.
  • Prepare for errors
  • Embrace change
  • Plan for redundancy
  • Think about future proofing
  • Look at (or study) biology or sociology how complex systems behave
  • Try your utmost to avoid mutable state. Go for stateless protocols (like REST and HTTP).
  • Functional programming might alleviate some of the pain

I guess I could go on and on. Very interesting subject :)

share|improve this answer
    
Look at (or study) biology or sociology how complex systems behave - C'mon. The rest of your answer was solid, but this has such a peripheral application to the problem described. –  Jim G. Feb 10 '11 at 21:46
    
@Jim G. Maybe. Biology won't help optimizing your for-loops, but if you want to come up with new perspectives, insights or effective abstractions (on software development), it does help to step out of one's sandbox. Arguing that biology (or sociology) has nothing to do with programming, it just few hops away from arguing that OOP or design patterns has nothing to do with programming. For example: OOP: biology -> Alan Kay -> OOP/Smalltalk. Or Design Patterns: sociology -> urban design -> Christopher Alexander -> A Pattern Language -> Design Patterns. –  Maglob Feb 11 '11 at 10:20
    
@Jim G. Cont. Some quotes, Alan Kay: "I thought of objects being like biological cells and/or individual computers on a network, only able to communicate with messages", and Wikipedia: "[Design Pattern] The idea was introduced by the architect Christopher Alexander in the field of architecture[1] and has been adapted for various other disciplines, including computer science" –  Maglob Feb 11 '11 at 10:20
    
Alright. I'm giving you a +1 for Try your utmost to avoid mutable state and other nuggets. My point is that if your manager tasked you with reducing complexity, you'd most certainly apply Occam's razor to the problem and get to work. I don't think that you or anyone else would "look to biology" for help with the immediate problem. –  Jim G. Feb 12 '11 at 17:05
add comment

I disagree with the spirit of @Péter Török's answer because it assumes that a team (or individual) can necessarily foresee the riskiest items early in the project lifecycle. For instance, in the OP's case, the team could not foresee the escalating complexity attached to the multi-threaded solution until their backs were against the wall.

The OP's question is a good one, and it speaks to a problem that many software development shops have.

Here is how I would deal with the problem:

  1. Follow Fred Brooks' advice and organize your developers like a surgery team.
  2. Choose a wise and "benevolent" master-surgeon that can both: A) Garner the trust and respect of his/her peers; and B) Make difficult decisions in a timely manner.
  3. Expect the master-surgeon to reduce complexity at the front-end and the back-end of the development process.

More on point #3:

  1. The master-surgeon should make a conscious effort to propose the simplest solution that will work. Years of meaningful experience should put the master-surgeon in a position to do so.
  2. The broader organization, that is the master-surgeon's superiors, should give the team sufficient time and resources to reduce complexity after the ship date. This will allow the development team to both ship code in a timely manner and perform kaizen to reduce complexity in an ongoing basis.
share|improve this answer
    
IMHO in the OP's case they should have started testing earlier, to uncover how (and if) their architecture works under real life circumstances. Btw by suggesting to have a "master surgeon", you seem to basically imply that there are people who can foresee the technical risks of the project - the exact point you claim to disagree with. –  Péter Török Feb 10 '11 at 16:11
    
@Péter Török: ...By suggesting to have a "master surgeon", you seem to basically imply that there are people who can foresee the technical risks of the project: No, I'm not. I'm saying that these people are both: A) Best suited to entirely avoid complexity in the first place; and B) Best suited to dig a team out of complexity after the code has shipped. –  Jim G. Feb 10 '11 at 16:17
    
IMHO we are talking about the same thing. The experience which helps your "master surgeon" choose the simplest solution which can possibly work is built up from memories of past projects and solutions, and knowing which solution worked (or didn't) in which specific case. In other words, (s)he looks through the applicable solutions for a specific problems and assesses the potential benefits and risks of each. This is what helps him/her pick the right one for the current situation, thus avoiding the riskier paths. –  Péter Török Feb 10 '11 at 16:30
    
@Péter Török: OK. Understood. Thanks. :) –  Jim G. Feb 10 '11 at 16:33
1  
This reminds me of a quote from the late, great, horse trainer Ray Hunt: "How do you get good judgement? Experience. How do you get experience? Bad judgement." –  glenatron Feb 10 '11 at 17:04
add comment

Code to interfaces

When writing new functionality interfacing with other functionality, make a boundary in form of an interface (the Java kind) through which all pass. This will

  1. ensure you have full control over what functionalities are used
  2. allow you to have multiple implementations of the same functionality.
  3. keep overall complexity down because the modules are only thinly connected instead of being fully intertwined.
share|improve this answer
add comment

one can start with a simple premise and model and suddenly have an escalation of complexity well into the development of the project

Not surprising.

This is software development. If you're not inventing something new, you're downloading an existing, proven solution.

There's little middle ground.

If you're inventing something new, then there must be at least one feature you don't fully understand. (To fully understand it, you'd have to have a working implementation, which you would just use.)

How to manage this?

  1. Have realistic expectations. You're inventing something new. There must be parts you don't understand.

  2. Have realistic expectations. If it seems to work right the first time, you've overlooked something.

  3. Have realistic expectations. If it was simple, someone else would have done it first, and you could simply download that solution.

  4. Have realistic expectations. You can't predict the future very well.

share|improve this answer
2  
Wait, so what you're saying is: Have realistic expectations? –  glenatron Feb 10 '11 at 15:45
add comment

Design and code with obsolescence in mind. Assume that what you code today will need to be cut out and replaced tomorrow.

share|improve this answer
add comment

The environment should be part of the specification. Thus a change to environment IS a change to specification. If, on the other hand, you based your prototype and design on an environment other than what was in the spec, you made a foolish mistake. Either way, you suck it up and find the least expensive way to implement the new complexity.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.