Ok I figured it out.
Answered here: How can I use Web.debug.config in the built-in visual studio debugger server?
So the config files are only combined when you publish, not when you are running against a local server. Pretty stupid IMO, when else would you ever use Web.Debug.config?
I will do as is suggested here: Use Visual Studio web.config transform for debugging
and just have Web.config as my default debugging config file, then have release for when releasing. Can't see a use for Web.Debug.config as this point.
Still, this is annoying because most of my settings I want to be set one way for all environments but when developing (eg customErrors On). This means I have to set them in Web.config for debugging, then in all my other environment configs change them.
Thanks everyone for responses.