I have been using an agile methodology (SCRUM) for about three years now and I see certain advantages to it, especially in the short-term feedback at many levels (from customers having early access to implemented features, from testers that can test features as soon as they are implemented, from other developers that can provide very early feedback on new code through review, etc).
On the other hand, I have two open problems, the first of which I will try to explain in this question.
Problem : Difficulty to get a good design
I try to do refactoring as soon as the code gets messy, I write unit tests as much as I can (which does help to prevent bugs in general and when refactoring in particular). On the other hand, developing some complex feature in small increments, with daily commits and continuously rethinking the code when it gets unstructured is not allowing me to produce really good design.
The only well-designed module I could produce recently I obtained by taking a different approach: I analyzed the problem for a few days (I actually had had the problem going through my mind for a couple of months before I started to work on it seriously), sketched a rather detailed design of all the involved classes and their relationships for a couple more days, and then locked myself up in my office and wrote down the whole code by working without interruption for about three weeks. The result was the best thing I have produced in a while, with very few bugs that were rather easy to locate and to fix, and with a very clear design, which has not required any relevant changes since then.
So up to now I found it much more effective to get an overall picture of what I want to do beforehand than starting to write code in small increments in the hope that the big picture magically emerges in the process. With my best effort, the small-increment development approach has always led me to worse design.
Question: Has anyone had a similar experience? Am I applying SCRUM the wrong way or what should I pay attention to if I want to develop in small increments and still end up with a well-designed piece of software? Or should I schedule a design user story before starting the actual coding? Is this considered a good practice, at least for features that are more complex than average?
EDIT - NOTE
I am aware of the fact that good design is not something absolute and does not have a value on its own but it depends on the context, and that one should aim at a design that is good enough for the problem at hand.
For example, I do not care (too much) about good design if I have to implement a simple component that (1) must be ready as soon as possible, (2) is going to be used only once, (3) is not used by other parts of the system (YAGNI).
I do care about good design when a component (1) will be used several times and in several different releases of a product, (2) needs to be maintained and extended over time, (3) has a lot of other components depending on it.