I do something similar to @Mud, but switch to previous view buffer, #, instead of the previous buffer in buffer list. Here is a binding key in my .vimrc:
nnoremap q :lcloseb#bd #
Close Location windows, if exist, switch to the previous view buffer, and then close the last switched buffer.