Here are two pictures.
on Lollipop:

on Pre-Lollipop:
So here it goes perfectly fine on Kitkat, samsung device to be precise.
I tried card_view:cardUseCompatPadding="true" but no avail. Didn't work!
Then I discovered from a stackoverflow post this
card_view:cardPreventCornerOverlap="false" and VOILA! Worked! There were no round corners (Since, I wanted none as the Card has an Image background).
The moral is, an extra padding is because of those tiny round corners which, need to be disabled. Basically that is not a flaw rather a Design constraint!
ORIGINAL SOF REPLY
Image: Notice that top corners are edges (A view which has colour and an image in background) while bottom has just TextViews and no backgrounds hence, the round corners. That means if a view is requesting match_parent inside the CardView, card_view:cardPreventCornerOverlap="false" will allow that to be taken up, on the affected corners.