If you treat refactoring as its own thing, rather than as part of the process of development, then it can become never ending.
If you follow the test driven development approach of red, green, refactor, then you typically won't have to spend a lot of time refactoring for two reasons:
You'll be refactoring as you go, so it shouldn't become a mammoth task, just another small step in the TDD cycle.
The internal "pressure" to get the next test written will keep your refactoring efforts to a sensible level.
If you are working with legacy code, then follow Uncle Bob's "Boy scout rule." Timebox yourself to a short time period (say, 30 minutes) and make your goal to check in the code in a cleaner state than it was before. Each time you have to touch the code, you will leave it slightly better. Over time, the most modified code will be the most readable and well factored in your codebase.