EDIT - SOLUTION:
I ended up figuring out a way to solve this issue. Because manually changing the height of the ImageView removes the extra padding,
Please set your view's LinearLayout attribute i.e.android:layout_height="match_parent". Hope it will work.
android:layout_height="match_parent"