Our team redesigns the legacy software and there are a lot of design meetings going on. Many of them are not very effective and lack good analysis of the proposed design. Is there any formal way to approach this kind of discussions to make them more effective? I heard about SWOT for example, but afaik it is used more for business projects.
What you probably want is to define a set of interfaces that speak your business story, all the while ignoring the existing code, plug them in with a minimum rewrite, and then slowly reimplementing more and more of these interfaces. The formality in the process ends there, but I hope its practicality is obvious.
I assume you want the redesigned system to behave exactly as the legacy system, at least in the beginning. This means to need to capture the concepts and behavior of the legacy system first as precisely as possible. I recommend to create architecture models and use them for discussion. Everybody should have a good understanding of the modeling language you use.
Check out the methology we use at SAP, which is described in this 2 part blog post: http://scn.sap.com/people/bernhard.groene/blog/2008/01/09/how-to-communicate-architecture-technical-architecture-modeling-at-sap-part-1
Don't just create the block diagrams. I recommend to start with entity relationship diagrams and also add definitions of the entities. Then do the block diagrams - don't forget to describe protocol boundaries, and add a short description what each block does. Last thing should be one of the dynamic diagrams. Use the other diagram types if you need them.
What you will experience are lively discussions e.g. if there is a 1:* or a 1:0..* relationship between two entities. Or you will learn a lot about dependencies between components if you try to figure out where you could establish a system boundary, and so on.
If you use vague powerpoint drawings of your legacy system architecture, everybody will have his own interpretation and understanding, so you won't make much progress, and your new system will probably not reflect the legacy system.
I blogged a couple of years ago about never rewrite a system. Basically like Strangler Application it is about inserting transparent facades in front of interfaces to the existing systems and incrementally grow this redirection to the new system until the old system is no longer in use.
And never re-writing the whole system in one go as that will take way too long with no business benefit at least for a long while. You need to deliver business value along with replacing the existing legacy system.
As for the actual estimations, you can apply many time consuming in-detail estimations using various techniques. However they will all just be as wrong as a finger in the air estimate, just now wrong in detail as Dan North would say.
Create a few prioritised epics of what features are needed initially, break them up further if needed but not much more. Then just throw some order of magnitude estimates of various parts of the job (t-shirt sizes, 1-10-100 story points or days or whatever). Applying StranglerApplication and then deliver some tracer bullet stories to get a much better idea of the actual complexity and viability of the project.