These two questions give similar algorthims for shuffling an IEnumerable:
The second version will probably be a bit slower because of RemoveAt. Lists are really arrays that grow when you add elements to them, and as such, insertion and removal in the middle is slow (in fact, MSDN states that RemoveAt has an O(n) complexity).
Anyway, the best would be to simply use a profiler to compare both methods.