Looking at the source I found out that setting a shadow extends the clip rectangle.
A trick is to set an invisible shadow just beyond the character.
For example:
android:shadowRadius="2"
android:shadowDx="2"
android:shadowColor="#00000000"
I think this solution is better as it will not extend the width of the TextView which may happen when adding an extra character (which is more apparent with a background).