Assume this model:
Public Class Detail
...
As an alternate way around this, I applied a RegularExpression attribute to catch the invalid entry and set my message there:
[RegularExpression(@"[0-9]*$", ErrorMessage = "Please enter a valid number ")]
This slightly a hack but this seemed preferable to the complexity the other solutions presented, at least in my particular situation.
EDIT: This worked well in MVC3 but it seems that there may well be better solutions for MVC4+.