I would like to break out my controllers and views into separate class libraries so they can be reused in multiple ASP.NET MVC 3 applications. The controllers part was not
Take a look at mvc contrib's portable areas:
http://www.lostechies.com/blogs/hex/archive/2009/11/01/asp-net-mvc-portable-areas-via-mvccontrib.aspx
They were made specifically for this purpose. If you go that road, it is less code you have to mantain ;-)