UPDATE: I thought it worked correctly. But after some test trouble still exists *sniff*
Then I made a simpler version to see what exactly happen and
It works! Just remove the transition from your fragment replacement, in my case I removed the following from my code:
.setTransition(FragmentTransaction.TRANSIT_FRAGMENT_OPEN)