In your particular case I would not modify the structure of the HashMap but merely null the value you want to remove. Then if you end up visiting a null value just skip it.
In the general case I prefer to use a Stack for things like this because they're particularly easy to visualise and so I tend to have less issues with border conditions (just keeping popping 'till empty).
