I would like to nest asp.net web app within sitecore site, something like this:
.
You should be able to create a virtual app under a Sitecore root path. Because its an app it will use its own app pool and you can have a web.config for it. I guess the physical files will be OUTSIDE of the Sitecore Website folder but the from a URL you can make the virtual app located anywhere below the Sitecore root.