I found another interesting approach, which is easier to code and debug than the expression tree solution I originally used:
http://www.codeproject.com/KB/cs/genericnumerics.aspx
This solution uses generic type constraints in an interesting way to ensure all required operations are supported, but without introducing any boxing or virtual method calls.