I had a discussion a few weeks back with some co-workers on refactoring, and I seem to be in a minority that believes \"Refactor early, refactor often\" is a good approach t
I refactor when:
I'm modifying code and I'm confused by it. If it takes me a while to sift it out, it needs refactoring.
I'm creating new code and after I've got it "working". Often times I'll get things working and as I'm coding I realize "Hey, I need to redo what I did 20 lines up, only with a few changes". At that point I refactor and continue.
The only thing that in my opinion should stop you from doing this is time constraints. Like it or not, sometimes you just don't have the time to do it.