How to groupBy an iterator without converting it to list in scala?

Question

Suppose I want to groupBy on a iterator, compiler asks to \"value groupBy is not a member of Iterator[Int]\". One way would be to convert iterator

Answer 1

As said in other responses, the only way to achieve a lazy groupBy on Iterator is to internally buffer elements. The worst case for the memory will be in O(n). If you know in advance that the keys are well distributed in your iterator, the buffer can be a viable solution.

The solution is relatively complex, but a good start are some methods from the Iterator trait in the Scala source code:

• The partition method that uses both the buffered method to keep the head value in memory, and two internal queues (lookahead) for each of the produced iterators.
• The span method with also the buffered method and this time a unique queue for the leading iterator.
• The duplicate method. Perhaps less interesting, but we can again observe another use of a queue to store the gap between the two produced iterators.

In the groupBy case, we will have a variable number of produced iterators instead of two in the above examples. If requested, I can try to write this method.

Note that you have to know the list of keys in advance. Otherwise, you will need to traverse (and buffer) the entire iterator to collect the different keys to build your Map.