UPDATE: I thought it worked correctly. But after some test trouble still exists *sniff*
Then I made a simpler version to see what exactly happen and
This issue seems to still be occurring in appcompat 22.1.1. Wrapping the SwipeRefreshLayout inside a FrameLayout solved this for me