I worked on a large project with 5 million + lines of code with 23 separate teams and ~250 people (developers and managers) where all the code was brought together into a monolithic build. The key was to continue decomposing parts into new teams/projects, creating clear interfaces, and creating meaningful abstractions. The most important part was the abstraction concept.
Think of a car. It doesn't really matter what type of car. There is a general idea of a car. Someone/team made the tires, the engine, the brakes, etc... Each of these items can also be an abstraction where you continue to decompose the the parts into other meaningful abstractions. This is basically a top-down approach to design. You're trying to create high level, manageable pieces and to have an understanding of what you're building before you start. Where this approach goes wrong is in trying to make/decompose the whole system before you start anything at all (analysis paralysis).
What you're doing is a bottom-up approach, where you start building and wait for the overall design to emerge. Most of us developers love bottom up because we get to start coding right away. Where I think you're finding the difficulty is in "seeing the forest from the trees", where it's hard to visualize the proper abstractions and how other pieces will fit together. That's why you are "afraid of breaking what I already have working in order to implement a new feature". You haven't thought enough about the whole before you started. This is where bottom up process seems to run into problems and, in my experience, leads to a bowl full of agile spaghetti.
Using the car as the example, you have made the tires, but now you're afraid to add the brakes because you're not sure how it will fit within the overall car.
The sweet spot is somewhere in the middle of these two approaches (i.e. you should use both). It's impossible to be fully top-down because you can't know what something looks like until you create it, so thinking you can decompose a system fully in this way is folly. Bottom-up has problems too because it leads to meandering through a project, poor early design choices, and generally poor interfaces. What I have found is that you start with the top-down as a guide to create abstractions and then use the bottom-up to build out on each abstraction. If you have the abstractions then you will be thinking about how they all fit together as you code.