url with extension not getting handled by routing

Question

I've been folowing the advice from this article for setting up a robots.txt file in asp.net mvc3 for using a controller to handle the server response, and IIS 8.0 express is returning a file not found error, rather than an asp.net error.

How do I get IIS to not look for a file in these cases? Is there something I need in the web.config?

Answer 1

IIS tries to be intelligent here. He intercepts the dot in the url and thinks that this is a static file and attempts to serve it with the default StaticFile handler. it dopesn't event get to the managed ASP.NET application.

The first possibility is to add the following in your web.config

<system.webserver>
    <modules runAllManagedModulesForAllRequests="true" />

but actually that's not something I would recommend you doing because this might have a negative effect on the performance of your application because now all requests to static files (such as .js, .css, images, ...) will go through the managed pipeline.

The recommended approach is to add the following handler to your web.config (<handlers> tag of <system.webServer>):

<system.webServer>
    <handlers>
        <add name="Robots-ISAPI-Integrated-4.0" path="/robots.txt" verb="GET" type="System.Web.Handlers.TransferRequestHandler" preCondition="integratedMode,runtimeVersionv4.0" />
        ...
    </handlers>
</system.webServer>

Notice how we have specified that this handler will only apply to a particular URL and HTTP verb.

Now when you GET /robots.txt, IIS will no longer handle it with the StaticFile handler but will instead pass it to the managed pipeline ASP.NET. And then it will be intercepted by the routing engine and routed to the corresponding controller action.

Answer 2

Unless you need a dynamically generated robots.txt file, which is very rarely necessary, just do the following:

• Ignore the route to robots.txt

  routes.IgnoreRoute("robots.txt");

• Add the robots.txt file to your root dir