I have an EditText I want to zoom it, and scroll with setScaleX/setScaleY and it works fine - text is being edited in the right position.
EditText
setScaleX
setScaleY
Did you try to apply different styles to these elements? I think it should work with some conditioning.
@drawable/text_select_handle_middle @drawable/text_select_handle_left @drawable/text_select_handle_right