I am always making small refactorings in my code. I know as long as I have my unit tests to verify that everything is still functioning properly afterward, I see no harm in doing it as I go. That way you don't get that vague "needs refactoring" feeling every time you work on it.
Now if it requires a large refactoring, it's best to plan for that and set aside some time.
