I finally found out: the issue can be solved by just calling the mViewPager.setCurrentItem(position) with an extra parameter to false, which is the smooth scroll for the ViewPager.
After this, the scroll will be done without any smoothing and thus the animations won't be seen.