When you clear your adapter and call notifyDataSetChanged() - StaggedGridLayoutManager manager still has mCachedStart/End in Span[] instances. So when you add new items to adapter - layout manager draws items with offset, and this is a huge problem.
EtsyStaggeredGridView has method resetToTop() which clears all cached stuff and draws everything from scratch.
There are some workarounds:
- Call
layoutManager.onDetachedFromWindow(recyclerView, null)after clearing an adapter - Set new layout manager which is bad
