I have a large enterprise application containing both WebForms and MVC pages. It has existing authentication and authorisation settings that I don\'t want to change.
I found myself with the same exact problem, the following article pointed me in the right direction: http://msdn.microsoft.com/en-us/library/aa479391.aspx
MADAM does exactly what you are after, specifically, you can configure the FormsAuthenticationDispositionModule to mute the forms authentication "trickery", and stop it from changing the response code from 401 to 302. This should result in your rest client receiving the right auth challenge.
MADAM Download page: http://www.raboof.com/projects/madam/
In my case, the REST calls are made to controllers (this is a MVC based app) in the "API" area. A MADAM discriminator is set with the following configuracion:
Then all you have to do is add the MADAM module to your web.config
Remember to add the valid sections to the web.config (SO didn't let me paste the code), you can get an example from the web project in the download.
With this setup any requests made to URLs starting with "API/" will get a 401 response instead of the 301 produced by the Forms Authentication.