It is related on how the LayoutManager calculates the sizes. Here you have the related bug with some workarounds people have used.
NOTE: With the release 23.2 of the support library now the wrap content is supported, so looks like they fixed it. You can checkout the changelog here