We know that IIS caches ConfigurationManager.AppSettings so it reads the disk only once until the web.config is changed. This is done for performance purposes.
Someo
As John says only spend more time on this if you are actually seeing a performance hit.
Also I'm pretty sure that these applications hold the configuration in memory, and to see any changes within a config the application would have to be restarted.
For some further reading about remaining mysteries.