I had a discussion a few weeks back with some co-workers on refactoring, and I seem to be in a minority that believes \"Refactor early, refactor often\" is a good approach t
I try to go by this motto: leave all the code you touch better than it was.
When I make a fix or add a feature I usually use that opportunity to do limited refactoring on the impacted code. Often this makes it easier to make my intended change, so it actually doesn't cost anything.
Otherwise, you should budget dedicated time for refactoring, if you can't because you are always fighting fires (I wonder why) then you should force yourself to refactor when you find making changes becomes much harder than it should and when "code smells" are just unbearable.