Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Software development techniques exist to solve problems. I think a key problem we face is conquering complexity. Also, software developers must often classify and understand complex systems, separating accidental complexity from essential complexity. I believe that sufficiently useful definitions of these terms all exist on Wikipedia.

My question is: What techniques are most valuable in conquering complexity, as a professional software developer, and/or software architect?

Answer examplar; a blog post on conquering complexity that seems to be coming at things from a java/c++/OOP centric perspective.

share|improve this question

closed as too broad by Telastyn, Scant Roger, kevin cline, Ixrec, gnat Jan 10 at 15:56

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

up vote 7 down vote accepted

YAGNI. The best way to avoid accidental complexity is to stop making stuff more generic and flexible than they have to be.

For instance, don't start looking for frameworks and libraries until you actually know that you need them. Instead of solving todays problems, we spend time thinking up potential problems that might arise in the future. Don't do that. Focus on today.

share|improve this answer
2  
I've bee thinking about this for about a week, and I really think this is the most obvious, and correct answer. A lot of accidental complexity gets in BECAUSE WE PUT IT IN THERE. So, sometimes, ya gotta just stop doing that. :-) – Warren P May 2 '11 at 19:36

I find Event Driven Architecture and Command-Query Responsibility Segregation to be the most common techniques I use to conquer complexity.

In a nutshell:

  • UI Controllers submit granular Commands on behalf of the user
  • Command Handlers mutate application state through subsystems (like a domain model, or simply transaction scripts)
  • Changes in application state raise events
  • Event handlers react by submitting more commands and/or interacting with application services (updating auxiliary data for display, sending emails, etc., etc. - a lot happens here and this is the main method of decoupling auxiliary logic from that logic that modifies the application state)

On a large scale, I try to stick somewhat rigidly to the send a command, handle the command, raise events, handle events pattern - it can lend large scale organization to a variety of project types.

Then, I allow handlers to achieve their function through whatever mechanism seems appropriate. These mechanisms form sub-components of the application like a domain model with persistence, loggers, email helpers, etc.

Allowing flexibility in the implementation of these sub-components enables agility (write it to get it done, if need be), code reuse (whether linked library or copy and paste), refactoring (let's base off of this previously written component but improve/change it as so).

But sticking to EDA & CQRS gives us some architectural consistency across projects, which makes navigating a foreign code base much easier. It also provides nice points to implement functionality with AOP - like authorizing & recording commands, persisting events, distributing workload, etc.

share|improve this answer
    
It should also allow components to be written in the language that suits them best. – Christopher Mahan Apr 25 '11 at 23:08

I found the following helpful to reduce complexity:

share|improve this answer
    
Especially think "Bottum Up design" is powerful, and using an expressive language (perhaps that means, high level). – Warren P Apr 26 '11 at 1:31

Hands-in the Pocket Explanations.

(The phrase comes from this: http://portal.acm.org/citation.cfm?id=70024

If you can't explain it with your hands in your pockets, it's too complex. Simplify until you can explain it.

It helps to summarize use cases, architectures, design patterns, programming idioms and the like as short, easy-to-grasp stories.

This usually means that you have to create meaningful chunks or abstractions tht have to be isolated and explained separately.

These chunks are not programming language monstrosities, but are actual useful simplifications. More like the "class" vs. "instance" nature of abstraction than the "abstract superclass" vs. "concrete superclass" problem where the OO mavens have gone crazy.

share|improve this answer

Coarser Objects With Fewer Interactions

For me complexity and the overwhelming feeling it causes tends to take on two extreme forms. Note that this is for large-scale codebases, where anything under a million lines of code is not very "large" to me (I once heard someone describe a project with 16k LOC as "large-scale", that seems extremely tiny to me, and something that could easily be maintained and easily comprehended in full by a single person, even if the code is pretty foul). I'll use somewhat of a SimCity analogy, and keeping it somewhat abstract and meta-level to promote a mindset to mitigate complexity, yielding multiple possible approaches, more than some absolute technique set in stone.

  1. Megalopolis, like an overpopulated city that is too hard to manage. This tends to apply towards the most inflexible systems that are indivisible, where everything feels like one gigantic implementation detail. This leads to implementation-level complexity at every level of the system.
  2. Disparate teeny villages whose trade routes form a complex interaction that causes our brains to overload just trying to map it out and think about the control flow involved. This leads to design-level complexity when examining the system as a whole.

To me a lot of the point where even a very well-engineered project can exceed one person's mental capacity to effectively reason about exactly what will happen in response to every user-end operation breaks down anywhere after about half a million to a million lines of code for a well-engineered case (not some horrible mess which might cause our brains to shut down at a fraction of that), give or take. At least that's when I find my ability to memorize every nook and cranny and know what will actually happen ceases to work and starts to become replaced by surprises. Note that I'm talking about bird's eye administrative level, not our ability to memorize every single line of code, but every single thing that will happen at some reasonably high-level perspective (every kind of relevant operation at some reasonable level, not too micro, semi-macro, like say everything relevant to make a design change that impacts one of those areas that gets brought up at a meeting when a design change is proposed, not going into individual lines of code). Yet there's a technique I've discovered that allows me to more easily comprehend much larger codebases than this without getting overwhelmed by the complexity associated with the scale, and I'll cover that later (calling it the "city" design).

Megalopolis Complexity

enter image description here

The system feels like one massive implementation detail. There are no clean barriers in between the code to effectively section it off, it's all one organic whole characterized by tight coupling all over the place and a glaring absence of generalized functionality that can actually be reused outside of the immediate context. With these types of codebases, you either understand everything or understand next to nothing ("What, this megalopolis has a Disney land right next to a strip club? What? Godzilla is attacking where?").

This system tends to be incredibly difficult to test because it's so indivisible. Nothing is independent except maybe a general-purpose toolkit library used to help implement it. I tend to find these impossible to comprehend in full at large scales, yet they necessitate it, making for a mystery scenario where half the changes often end up having unanticipated repercussions.

Teeny Villages/Islands

enter image description here

Here, while each little village is so simple and modest, the interactions between them overwhelm the brain. Of course there were abstract interfaces between each village, all communication was done through abstractions and DI, but to me it doesn't really help with "complexity" to do that, only "flexibility" to, say, swap one village with another without breaking anything. Some of these dependencies also are the result of villaging observing other villages, where the arrows don't even indicate a direct function call of any sort or even a real dependency, but merely an event triggered from one village which indirectly calls another.

This system tends to be hard to test effectively outside of a very coarse integration testing mindset because a lot of the problems that occur will be through the complex interaction of very simple objects (ex: only through a precise subscription of a series of cascading events). The objects, being simple, will often do a wonderful job of maintaining their own invariants. The invariants that will be difficult to maintain are broad, system-level invariants ("this village should not be having a festival while this other village is trying to negotiate a trade agreement with it as a result of an event that occurred in village C").

These are the types of systems that tend to emerge when we embrace engineering principles to some peak level, only to find a new kind of complexity arise in terms of interactions and the ultra-granular level of simplicity for each module.

My tendency was to err towards number 2, the teeny village style, which made it so each object was very simple, very reusable, a hammer here, a screwdriver there, a bandsaw here, etc., and all unified under a central set of abstract interfaces forming an SDK (Hammer implements ITool and IBluntWeapon or something like that). Even my projects were simple and always bundling ultra-highly-related functionality that add up to one very narrow (but widely applicable) cohesive responsibility, SRP taken to the max.

The problem was that when I looked at the system from an overseer perspective, what I found overwhelming was the interactions between all these little granular objects, each with such simple implementations. It then became almost as hard to reason about system behavior as the megalopolis, with recursive events and things notifying things notifying things, all loosely-coupled and highly cohesive, but the control flow going all over the place. The objects became very simple, little islands triggering events that would be broadcasted to other little islands, but the interactions that went on between them became incredibly complex. The megalopolis makes objects extremely complex and kind of lumped together into one giant, indivisible subsystem with everything tightly-coupled together, but the interactions between them potentially simpler.

It's like a puzzle where each puzzle piece is a dot and we're trying to fit them all together and make sense out of it. One might call it the difference between enormous implementation complexity and enormous complexity in the communication between interfaces.

City Complexity

enter image description here

So there was something I found to relieve complexity, and that is to seek something in the middle of these two, the "city" -- chunky enough puzzle pieces that we'd only need a couple dozen to form a complete puzzle. The result was coarser objects, higher-level interfaces with more complex implementations behind them, but fewer objects, and objects which often modeled collections of things rather than individual teeny things. If there were teeny objects, they'd either be implementation details or a part of a totally generalized and independent library with no interactions between the objects and no generalized event system.

I found this helped the most, since my tendency was to err on #2 with complex interactions of observers. I "flattened" the system, sat on it and squashed it flat, made implementations more complex in favor of fewer interactions, but not so complex that I ended up with a megalopolis design.

Processing then took on a bulkier, loopy fashion, with no dependency order which could screw things up (each city was doing enough work, and there were not too many cities, to reason about what would happen if we started changing the order in which each city would do its processing).

Balancing Act

In general two simple objects are easier to reason about in terms of complexity and twice as flexible as one object twice as complex as either of them. But there's some kind of balancing act here to be had when we zoom out of an individual, simple object and look at the whole architecture. An example is an entity-component system which thrusts all the complex implementation into coarse systems, and avoids complexity at the granular entity and component level (turning an entity into a mere collection of components, and components into mere data, while systems processing entities in loops then take on all the complexity). The systems in the ECS design then become the most complex parts of the codebase (the only parts that contain functionality), but are few in number and makes it much easier to manage intellectual complexity when we just have a handful of centralized places that actually do anything, while the rest is just data belonging to components and combined to form entities.

The entity-component system is a perfect example of that kind of "city" graph I consider the ideal to strive towards, allowing it to grow to a substantial scale without overloading any one individual's brain trying to comprehend everything. It's not always applicable, but there are other ways to achieve the kind of "city" design of moderate-complexity objects doing most of the work.

These kind of "balanced city" designs accept a little more complexity of implementation upfront in exchange for simpler interactions and fewer systems making up the bigger picture. It doesn't achieve complexity through the layering of very simple pieces with each layer being ultra simple. Instead it accepts a slightly flatter design with moderately-complex layers of functionality in favor of fewer layers (ex: 25 more complex Photoshop layers, not 1600 uber-simple Photoshop layers).

Event-Driven Programming

I don't think event-driven programming reduces complexity. It raises flexibility tremendously when we can attach any function to be called by any event. But it also makes it so, when some event occurs like some state change event, what actual code gets executed in advance? In a complex codebase written by a team, it's usually a giant mystery until we start tracing through it in a debugger and drawing out the resulting graph. It's often what makes all these teeny little dots making up puzzle pieces cease to be aggregated and combined into some bigger concept, when the dots can all live in some space and trigger events instead with anyone capable of subscribing.

enter image description here

To me complexity ultimately lies in the unknown, what is too much relevant information for our brains to grasp (as in either the megalopolis or teeny-but-numerous villages case) and leaving gaps of unknowns, of mystery. Event-driven programming can impose such gaps of knowledge even at smaller scales, since they model a mechanism by which we don't know what functions are going to be called by looking at the code surrounding the site in which an event is triggered.

So I actually think event-driven programming and push paradigms, applied heavily, is one of the quickest ways to increase, rather than decrease, complexity, unless your idea of complexity does not require you to know which functions actually get called in the system (perhaps this would be true in a system that largely favored immutable data, and allowed code to be executed across functions in any permutation possible without compromising correctness). An alternative is if we're working with a visual programming kind of thing where we can always see what event handlers are hooked up to an event through the form of a graph, always available on demand (I often dreamed of such tools when dealing with complex event-handling).

Events and observer-style designs are often a fundamental necessity, but I found it tremendously helpful to use events sparingly, and to avoid cascading events whenever possible (an event leading to an operation which triggers another event). Striving for shallow call stacks tends to reduce the feeling of complexity to me dramatically. I've worked in systems where the call stacks would sometimes be over 40 levels deep, with a dozen recursive events triggered in between, and that was the epitome of "complexity" to me.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.