I dont see anthing wrong with the code you posted. The only thing weird to me is this line
setSelected(fileHolder.itemView, mSelectedArray.get(i));
in your onBindViewHolder method in the adapter.. are you updating this array too when you change the size of your list of items in the array?