If so, at what stage? or it is also an iterative process? If iterative, how formal? Also, how useful is UML for a solo programmer doing open source? While I like to see a picture of the system, it feel so formal and a bunny trail for TDD.
The use of TDD does not prevent the use of UML, nor does the use of UML prevent TDD. In fact, I would suspect that using UML as a design tool to capture the behavior and interactions of the system would make it easier to write tests first. You would have a reference model of the implementation to use. However, as you implement, you would also need to keep your UML up-to-date, or it will quickly lose its value.
If you are going to apply UML in a project, there are three points at which it would be most useful.
The first would be when designing the architecture of the system, where you would probably want to use component, deployment, and communication diagrams to show how large pieces of the system will be fitting together and where each component will be deployed. A deployment diagram would be especially useful in a distributed system, where you have nodes will different responsibilities and capabilities.
The next point would be a more detailed design phase, when you start planning class and method level interactions. Here, you would probably find class, composite structure, sequence, object, and state diagrams most useful. The specific diagrams that you would need depend a lot on the complexity of the system and what you want to show.
At each of these two phases, the use of UML allows for you to carry out design reviews. You can validate your design against your requirements to ensure that you aren't missing anything. In a TDD environment, you might want to use these reviews to identify potential problems with implementation of your tests.
The final stages would be during development, maintenance, and enhancement. You have a captured architecture and design that you can refer to. This should also capture your reasoning. As you continue to grow your system, you can see your old design decisions along with why you made them, to determine the feasibility and extent of changes of adding new features. It also provides a blueprint to work from when developing the system - you've done at least some of the thinking up front, so you don't need to worry about connecting your modules on the fly.
Iterative modeling is definitely something to strive for. Not only should you revisit your models early in each iteration as you are designing and planning for new features, but you should also look at reverse engineering to try to keep your models up-to-date. Not all tools can automatically reverse engineer all diagram types, but you can try to keep the other ones up-to-date manually. If you do let a diagram become outdated, be sure to denote it or nix it from any repositories or documents so you don't make decisions based on invalid data.
I tend to be more informal with my UML models. I use the notation that I need to convey to myself and others exactly what the system is doing. I tend to be more minimalistic, focusing on getting a particular message to myself and other engineers for use when developing the system. I capture the message in text (such as notes on the diagram or by embedding the diagram in a Word document along with explanatory text) The idea is that as I develop the system, I'm also developing documentation for myself and others.
If it's a small system that's going to be built pretty quickly (<6 months of consistent development, or so, just as an off-the-cuff estimate) by one or two developers, the impact of UML would probably be minimal. Over a longer period of time, for more complex systems, or with a larger team, it will help you capture architectural and design decisions and then communicate these across the development team.
The point of UML is not about you, but about communication. If and when other people join your project as testers or developers, having up-to-date UML models can help them understand them system easier. A faster understanding means they are more easily able to get up to speed and start contributing, a plus for everyone.
Of course you can! You can use UML if and when you feel it would be helpful. It is just a tool and as such can be used not matter what development lifecycle/techniques you are using.
With TDD (and agile, which is usually not far away) it would be useful if, when implementing a new feature, a visualization of the existing code-base would help you or you want to visualize the changes/additions of your new feature. It should not be used to 'define' the code-base in this context, but only to help you understand it. You might just draw it freehand on a piece of paper. Agile/TDD development is quite fluid and a laminated UML diagram would no-doubt be thought of not unlike the devil.
Your question reflects the perceived chasm between "waterfall" and "agile" thinking, where UML is considered a technique used in BDUF (Big Design Up Front), which is an absolute NO in agile circles.
However, the agile manifesto never stipulated that one should not do any documentation or any design or any architecture up front. It only says "Just enough" and "we value working code more than documentation".
There are projects which do require a bit of up-front design and architecture, and UML is an obvious choice when trying to come to grips with these issues. The trick is to use it "just enough". How much exactly "just enough" is, depends on the project and the team, so you'll have to figure that out for yourself, I am afraid.