Refactoring is - and should be - an ongoing process. It's not enough to simply meet requirements with a working and tested implementation that is still a little incomplete.
"Make it work, then make it work better".
I can't remember where I read that quote, but this is the key to applying refactoring well, and I count it as unprofessional to do otherwise.
Continuous refactoring is like wiping up the spills while you are cooking, and cleaning your dishes after you've eaten your meal. Targeted refactoring is like finding a dirty kitchen, but only having the time to wash a dirty glass or two. Would you rather live with a continuously dirty kitchen, or would you prefer to keep things clean as you go along?
You get the code to work, then you refactor your code to ensure that you have the best implementation you can possibly use. If you're doing something familiar, it may be that you implement the best code first time, however it bears taking a moment to double-check your work to be sure. If it looks as though you could improve your code, then you try to refactor to make sure your code is at the very least as lean and clean as you can make it. This means you are reducing the amount of technical debt you leave behind, and you make it easier to read and refactor the next time the code needs to be dealt with. This is the core value behind the TDD mantra "Red-Green-Refactor", except that where in TDD you refactor primarily to remove duplication, it pays to also review other items that could be refactored, such as large classes, long methods, and other "code-smells" that can often contribute to technical debt.
If you find yourself facing a major redesign, then perhaps you can put it off for a while, particularly if you are running very low on time in your schedule. This is however provided the functionality of your code will not be compromised, and also provided the implementation will continue to meet the requirements. This sort of situation should be a rare occurrence, and you can help to ensure it is even rarer if you are continuously refactoring as you go along. Even more important however is that you can't risk leaving your major changes for too long, otherwise you will end up creating an even bigger workload later which could either be much more costly to fix, or could end up resulting in an even more costly project failure.
I get the impression that many people tend to confuse the definitions for Refactoring and Re-engineering. The two terms describe strategies to manage very different situations. If you wish to re-engineer, you're making a commitment to make a drastic change which will alter the behaviour of a system. This will invalidate some tests, and will also require new tests. When you Refactor, you are ensuring you're system continues to behave exactly the same as it did before the change, however you are also ensuring that your code will have longevity, and that it will be easier to maintain over time. You're not "pimping" your code for the hell of it, you are committing to a professional standard of clean code that will reduce the risk of failure, and will ensure your code remains a pleasure to work with, and of a professional standard.
Going back to the broken windows analogy, if you break the window you should repair it right away. If you haven't noticed that a window is broken, then you need to decide the cost to you if you leave the window broken. Now, repeat the previous two sentences, but substitute Bug for window. You end up needing a different strategy. If you have created a bug as you code, you fix it right away, or you see if the changes will require a re-engineering effort, and you make a commercial decision as to when it will be best to sort the problem out. So you don't refactor to fix a problem, you refactor to ensure it is easier to find and fix problems. I don't care how amazing you think your code is, complex systems will always have problems that will need to be dealt with over time. This is what technical debt is all about, and why refactoring needs to be an ongoing process as you implement your code, and not left for some arbitrary future time.
So in short, the answer that it may at rare times be acceptable to defer major changes to code in order to make a deadline, however it should not be considered normal practice to treat refactoring as an exercise independent of your daily implementation work, and certainly never used as an excuse by teams unfamiliar with the code base as an option to avoid ensuring that their implementation is as lean and clean as they can possibly make it under the circumstances.