Coarser Objects With Fewer Interactions
For me complexity and the overwhelming feeling it causes tends to take on two extreme forms. Note that this is for large-scale codebases, where anything under a million lines of code is not very "large" to me (I once heard someone describe a project with 16k LOC as "large-scale", that seems extremely tiny to me, and something that could easily be maintained and easily comprehended in full by a single person, even if the code is pretty foul). I'll use somewhat of a
SimCity analogy, and keeping it somewhat abstract and meta-level to promote a mindset to mitigate complexity, yielding multiple possible approaches, more than some absolute technique set in stone.
- Megalopolis, like an overpopulated city that is too hard to manage. This tends to apply towards the most inflexible systems that are indivisible, where everything feels like one gigantic implementation detail. This leads to implementation-level complexity at every level of the system.
- Disparate teeny villages whose trade routes form a complex interaction that causes our brains to overload just trying to map it out and think about the control flow involved. This leads to design-level complexity when examining the system as a whole.
To me a lot of the point where even a very well-engineered project can exceed one person's mental capacity to effectively reason about exactly what will happen in response to every user-end operation breaks down anywhere after about half a million to a million lines of code for a well-engineered case (not some horrible mess which might cause our brains to shut down at a fraction of that), give or take. At least that's when I find my ability to memorize every nook and cranny and know what will actually happen ceases to work and starts to become replaced by surprises. Note that I'm talking about bird's eye administrative level, not our ability to memorize every single line of code, but every single thing that will happen at some reasonably high-level perspective (every kind of relevant operation at some reasonable level, not too micro, semi-macro, like say everything relevant to make a design change that impacts one of those areas that gets brought up at a meeting when a design change is proposed, not going into individual lines of code). Yet there's a technique I've discovered that allows me to more easily comprehend much larger codebases than this without getting overwhelmed by the complexity associated with the scale, and I'll cover that later (calling it the "city" design).
The system feels like one massive implementation detail. There are no clean barriers in between the code to effectively section it off, it's all one organic whole characterized by tight coupling all over the place and a glaring absence of generalized functionality that can actually be reused outside of the immediate context. With these types of codebases, you either understand everything or understand next to nothing ("What, this megalopolis has a Disney land right next to a strip club? What? Godzilla is attacking where?").
This system tends to be incredibly difficult to test because it's so indivisible. Nothing is independent except maybe a general-purpose toolkit library used to help implement it. I tend to find these impossible to comprehend in full at large scales, yet they necessitate it, making for a mystery scenario where half the changes often end up having unanticipated repercussions.
Here, while each little village is so simple and modest, the interactions between them overwhelm the brain. Of course there were abstract interfaces between each village, all communication was done through abstractions and DI, but to me it doesn't really help with "complexity" to do that, only "flexibility" to, say, swap one village with another without breaking anything. Some of these dependencies also are the result of villaging observing other villages, where the arrows don't even indicate a direct function call of any sort or even a real dependency, but merely an event triggered from one village which indirectly calls another.
This system tends to be hard to test effectively outside of a very coarse integration testing mindset because a lot of the problems that occur will be through the complex interaction of very simple objects (ex: only through a precise subscription of a series of cascading events). The objects, being simple, will often do a wonderful job of maintaining their own invariants. The invariants that will be difficult to maintain are broad, system-level invariants ("this village should not be having a festival while this other village is trying to negotiate a trade agreement with it as a result of an event that occurred in village C").
These are the types of systems that tend to emerge when we embrace engineering principles to some peak level, only to find a new kind of complexity arise in terms of interactions and the ultra-granular level of simplicity for each module.
My tendency was to err towards number 2, the teeny village style, which made it so each object was very simple, very reusable, a hammer here, a screwdriver there, a bandsaw here, etc., and all unified under a central set of abstract interfaces forming an SDK (
IBluntWeapon or something like that). Even my projects were simple and always bundling ultra-highly-related functionality that add up to one very narrow (but widely applicable) cohesive responsibility, SRP taken to the max.
The problem was that when I looked at the system from an overseer perspective, what I found overwhelming was the interactions between all these little granular objects, each with such simple implementations. It then became almost as hard to reason about system behavior as the megalopolis, with recursive events and things notifying things notifying things, all loosely-coupled and highly cohesive, but the control flow going all over the place. The objects became very simple, little islands triggering events that would be broadcasted to other little islands, but the interactions that went on between them became incredibly complex. The megalopolis makes objects extremely complex and kind of lumped together into one giant, indivisible subsystem with everything tightly-coupled together, but the interactions between them potentially simpler.
It's like a puzzle where each puzzle piece is a dot and we're trying to fit them all together and make sense out of it. One might call it the difference between enormous implementation complexity and enormous complexity in the communication between interfaces.
So there was something I found to relieve complexity, and that is to seek something in the middle of these two, the "city" -- chunky enough puzzle pieces that we'd only need a couple dozen to form a complete puzzle. The result was coarser objects, higher-level interfaces with more complex implementations behind them, but fewer objects, and objects which often modeled collections of things rather than individual teeny things. If there were teeny objects, they'd either be implementation details or a part of a totally generalized and independent library with no interactions between the objects and no generalized event system.
I found this helped the most, since my tendency was to err on #2 with complex interactions of observers. I "flattened" the system, sat on it and squashed it flat, made implementations more complex in favor of fewer interactions, but not so complex that I ended up with a megalopolis design.
Processing then took on a bulkier, loopy fashion, with no dependency order which could screw things up (each city was doing enough work, and there were not too many cities, to reason about what would happen if we started changing the order in which each city would do its processing).
In general two simple objects are easier to reason about in terms of complexity and twice as flexible as one object twice as complex as either of them. But there's some kind of balancing act here to be had when we zoom out of an individual, simple object and look at the whole architecture. An example is an entity-component system which thrusts all the complex implementation into coarse systems, and avoids complexity at the granular entity and component level (turning an entity into a mere collection of components, and components into mere data, while systems processing entities in loops then take on all the complexity). The systems in the ECS design then become the most complex parts of the codebase (the only parts that contain functionality), but are few in number and makes it much easier to manage intellectual complexity when we just have a handful of centralized places that actually do anything, while the rest is just data belonging to components and combined to form entities.
The entity-component system is a perfect example of that kind of "city" graph I consider the ideal to strive towards, allowing it to grow to a substantial scale without overloading any one individual's brain trying to comprehend everything. It's not always applicable, but there are other ways to achieve the kind of "city" design of moderate-complexity objects doing most of the work.
These kind of "balanced city" designs accept a little more complexity of implementation upfront in exchange for simpler interactions and fewer systems making up the bigger picture. It doesn't achieve complexity through the layering of very simple pieces with each layer being ultra simple. Instead it accepts a slightly flatter design with moderately-complex layers of functionality in favor of fewer layers (ex: 25 more complex Photoshop layers, not 1600 uber-simple Photoshop layers).
I don't think event-driven programming reduces complexity. It raises flexibility tremendously when we can attach any function to be called by any event. But it also makes it so, when some event occurs like some state change event, what actual code gets executed in advance? In a complex codebase written by a team, it's usually a giant mystery until we start tracing through it in a debugger and drawing out the resulting graph. It's often what makes all these teeny little dots making up puzzle pieces cease to be aggregated and combined into some bigger concept, when the dots can all live in some space and trigger events instead with anyone capable of subscribing.
To me complexity ultimately lies in the unknown, what is too much relevant information for our brains to grasp (as in either the megalopolis or teeny-but-numerous villages case) and leaving gaps of unknowns, of mystery. Event-driven programming can impose such gaps of knowledge even at smaller scales, since they model a mechanism by which we don't know what functions are going to be called by looking at the code surrounding the site in which an event is triggered.
So I actually think event-driven programming and push paradigms, applied heavily, is one of the quickest ways to increase, rather than decrease, complexity, unless your idea of complexity does not require you to know which functions actually get called in the system (perhaps this would be true in a system that largely favored immutable data, and allowed code to be executed across functions in any permutation possible without compromising correctness). An alternative is if we're working with a visual programming kind of thing where we can always see what event handlers are hooked up to an event through the form of a graph, always available on demand (I often dreamed of such tools when dealing with complex event-handling).
Events and observer-style designs are often a fundamental necessity, but I found it tremendously helpful to use events sparingly, and to avoid cascading events whenever possible (an event leading to an operation which triggers another event). Striving for shallow call stacks tends to reduce the feeling of complexity to me dramatically. I've worked in systems where the call stacks would sometimes be over 40 levels deep, with a dozen recursive events triggered in between, and that was the epitome of "complexity" to me.