In my asp.net web site I have custom error pages defined as following in my web.config file.
After Googling about this issue, it seems that this is the default behavior that Microsoft ASP.NET provides for the situation. This is very bad for SEO. A work around I found is to check whether the requested file exists in an HTTP handler (or global.asax file), or use:
If the requested file does not exist, then rewrite the request path to a file not found page (if using an HTTP handler or global.asax), clear the server errors on the 404 error page code behind, and add a 404 error header to the response manually rather than waiting for server to do so.
Server.ClearError();
Response.Status = "404 Not Found";
Response.StatusCode = 404;