First, I realize this question may by somewhat long and vague and I apologize for this. This is probably a basic problem with a short name to anybody who's "got it", but as I find myself lacking in this regard, please bear with me in describing the problem.
I've been doing programming in this way or the other since I was about 11 years old. This means that I've been mostly teaching myself everything from the start. I received a technical education, but not strictly in Computer Science (I graduated with a degree in Photonic Engineering). We had programming courses of course, but this was mostly basic stuff for me and I didn't learn much new things. I've kept educating myself along the way for the joy of it and always knew I would pursue a career in programming, but all my projects were quite small at that time. I had no trouble keeping them in my mind and maintaining them.
Now, I find myself the lead on a team, but not in a corporate environment - I work for the university developing scientific software (in C++) for engineering applications. Suddenly the project is growing (relatively) big and I've trouble wrapping my mind around it most of the time. I'm losing a lot of time and effort on two things mostly:
- When I have to return to a section of code I've not worked on for a while, I have difficulty remembering how it worked. I spend a lot of time reviewing the header files for the relevant classes and reading the comments I placed along the way in the source files. I wish there was some form of "schematic" I could glimpse at and regain the picture more easily;
- When I introduce changes, sometimes I realize half-way that what I'm trying to do will break things somewhere else (or worse, it shows up only at runtime as a surprise). I revert and start doing it differently, only to find out I neglected the influence on some other component. I wish there was some "architecture diagram" where I could see how things get done, how what I'm trying to do will influence other components and a way for me to plan in detail before I start implementing changes.
Most of the people I work with have similar stories as my own - strong technical orientation and sometimes great skills, but with no way of organizing their work. However, their projects are usually much smaller than mine so they cope somehow. Anyway, what that means for me is that I'm on my own and I have nobody to learn the good practices from.
I took up a postgraduate course in managing IT and while I find it quite satisfying, it's mostly targeted at non-programmers, teaching about project management methodologies, budget/schedule estimations, enterprise architecture etc. - not software design and planning as such. That's OK, I'm trying to learn that stuff too. Of course, some tools (like UML) and the types of sofware development processes (cascade, iterative, agile...) were introduced, but obviously not in great detail and I've a hard time deciding what I should pick and use (and to what extent).
I've been reading many questions and answers about software design on SO - there are many about doing it using this or that particular tool or methodology, and if I was convinced that UML documentation would solve my problems - I'd pick it up and start using it. But some people swear by it, others say it's useless. I'm looking for an answer on a higher level of abstraction - are there ways to solve the two problems I'm having, and how do you personally do it? What should I learn to be able to do it, possibly without being bound to one particular tool? These come and go out of style from time to time, and I expect their applicability varies depending on the type of the project.
Thanks a lot for reading, I was unable to say what I mean more briefly (lacking in software design experience and vocabulary).