I tried many different combinations before I figured this out, but this appears to work correctly:
setRawInputType(InputType.TYPE_CLASS_NUMBER);
The key lies in the description for setRawInputType(int):
Directly change the content type integer of the text view, without
modifying any other state.