Here are two pictures.
on Lollipop:

on Pre-Lollipop:
If the Android 4 (pre-lollipop) behavior is desired, adding app:cardUseCompatPadding="true" to the CardView should fix it.
If the Android 5+ behavior is desired (which is actually the right behavior of card views according to material guidelines), the exact same thing can not be easily achieved. I usually use this fix to avoid defining multiple layout files and have reasonable output on all devices:
and in normal values/dimens.xml file we should have:
-8dp
-12dp
and in values-v21/dimens.xml:
0dp
0dp
note that the numbers -8dp and -12dp might have to be adjusted for your layout, since they depend on elevation, etc.
This is only a work-around to avoid ugly paddings in Android 4 views, without using different views in different layout files (which usually makes the code more difficult to maintain)