问题
Suppose, I'm writing a function to find "repeated words" in a text file. For example, in aaa aaa bb cc cc bb dd repeated words are aaa and cc but not bb, because two bb instances don't appear next to each other.
The function receives an iterator and returns iterator like that:
def foo(in: Iterator[String]): Iterator[String] = ???
foo(Iterator("aaa", "aaa", "bb", "cc", "cc", "bb")) // Iterator("aaa", "cc")
foo(Iterator("a", "a", "a", "b", "c", "b")) // Iterator("a")
How would you write foo ? Note that the input is huge and all words do not fit in memory (but the number of repeated words is relatively small).
P.S. I would like also to enhance foo later to return also positions of the repeated words, the number of repetitions, etc.
回答1:
UPDATE:
OK then. Let specify bit what you want:
input | expected
|
a |
aa | a
abc |
aabc | a
aaabbbbbbc | ab
aabaa | aa
aabbaa | aba
aabaa | aa
Is it true? If so this is working solution. Not sure about performance but at least it is lazy (don't load everything into memory).
//assume we have no nulls in iterator.
def foo[T >: Null](it:Iterator[T]) = {
(Iterator(null) ++ it).sliding(3,1).collect {
case x @ Seq(a,b,c) if b == c && a != b => c
}
}
We need this ugly Iterator(null) ++ because we are looking for 3 elements and we need a way to see if first two are the same.
This is pure implementation and it has some advantages over imperative one (eg. in other answers). Most important one is that it is lazy:
//infinite iterator!!!
val it = Iterator.iterate('a')(s => (s + (if(Random.nextBoolean) 1 else 0)).toChar)
//it'll take only as much as needs to take this 10 items.
//should not blow up
foo(it).take(10)
//imperative implementation will blow up in such situation.
fooImp(it).take(10)
here are all implementations from this and other posts seen in this topic: https://scalafiddle.io/sf/w5yozTA/15
WITH INDEXES AND POSITIONS
In comment you have asked if it would be easy to add the number of repeated words and their indices. I thought about it for a while and i've made something like this. Not sure if it has great performance but it should be lazy (eg. should work for big files).
/** returns Iterator that replace consecutive items with (item, index, count).
It contains all items from orginal iterator. */
def pack[T >: Null](it:Iterator[T]) = {
//Two nulls, each for one sliding(...)
(Iterator(null:T) ++ it ++ Iterator(null:T))
.sliding(2,1).zipWithIndex
//skip same items
.filter { case (x, _) => x(0) != x(1) }
//calculate how many items was skipped
.sliding(2,1).collect {
case Seq((a, idx1), (b, idx2)) => (a(1), idx1 ,idx2-idx1)
}
}
def foo[T >: Null](it:Iterator[T]) = pack(it).filter(_._3 > 1)
OLD ANSWER (BEFORE UPDATE QUESTION)
Another (simpler) solution could be something like this:
import scala.collection.immutable._
//Create new iterator each time we'll print it.
def it = Iterator("aaa", "aaa", "bb", "cc", "cc", "bb", "dd", "dd", "ee", "ee", "ee", "ee", "ee", "aaa", "aaa", "ff", "ff", "zz", "gg", "aaa", "aaa")
//yep... this is whole implementation :)
def foo(it:Iterator[String]) = it.sliding(2,1).collect { case Seq(a,b) if a == b => a }
println(foo(it).toList) //dont care about duplication
//List(aaa, cc, dd, ee, ee, ee, ff)
println(foo(it).toSet) //throw away duplicats but don't keeps order
//Set(cc, aaa, ee, ff, dd)
println(foo(it).to[ListSet]) //throw away duplicats and keeps order
//ListSet(aaa, cc, dd, ee, ff)
//oh... and keep result longer than 5 items while testing.
//Scala collections (eg: Sets) behaves bit diffrently up to this limit (they keeps order)
//just test with bit bigger Sequences :)
https://scalafiddle.io/sf/w5yozTA/1
(if answer is helpful up-vote please)
回答2:
Here is a solution with an Accumulator:
case class Acc(word: String = "", count: Int = 0, index: Int = 0)
def foo(in: Iterator[String]) =
in.zipWithIndex
.foldLeft(List(Acc())) { case (Acc(w, c, i) :: xs, (word: String, index)) =>
if (word == w) // keep counting
Acc(w, c + 1, i) :: xs
else
Acc(word, 1, index) :: Acc(w, c, i) :: xs
}.filter(_.count > 1)
.reverse
val it = Iterator("aaa", "aaa", "bb", "cc", "cc", "bb", "dd", "aaa", "aaa", "aaa", "aaa")
This returns List(Acc(aaa,2,0), Acc(cc,2,3), Acc(aaa,4,7))
It also handles if the same word has another group with repeated words.
And you have the index of the occurrences as well as the count.
Let me know if you need more explanation.
回答3:
Here's a solution that uses only the original iterator. No intermediate collections. So everything stays completely lazy and is suitable for very large input data.
def foo(in: Iterator[String]): Iterator[String] =
Iterator.unfold(in.buffered){ itr => // <--- Scala 2.13
def loop :Option[String] =
if (!itr.hasNext) None
else {
val str = itr.next()
if (!itr.hasNext) None
else if (itr.head == str) {
while (itr.hasNext && itr.head == str) itr.next() //remove repeats
Some(str)
}
else loop
}
loop.map(_ -> itr)
}
testing:
val it = Iterator("aaa", "aaa", "aaa", "bb", "cc", "cc", "bb", "dd")
foo(it) // Iterator("aaa", "cc")
//pseudo-infinite iterator
val piIt = Iterator.iterate(8)(_+1).map(_/3) //2,3,3,3,4,4,4,5,5,5, etc.
foo(piIt.map(_.toString)) //3,4,5,6, etc.
回答4:
It's some complex compare to another answers, but it use relatively small additional memory. And probably more fast.
def repeatedWordsIndex(in: Iterator[String]): java.util.Iterator[String] = {
val initialCapacity = 4096
val res = new java.util.ArrayList[String](initialCapacity) // or mutable.Buffer or mutable.Set, if you want Scala
var prev: String = null
var next: String = null
var prevEquals = false
while (in.hasNext) {
next = in.next()
if (next == prev) {
if (!prevEquals) res.add(prev)
prevEquals = true
} else {
prevEquals = false
}
prev = next
}
res.iterator // may be need to call distinct
}
回答5:
You could traverse the collection using foldLeft with its accumulator being a Tuple of Map and String to keep track of the previous word for the conditional word counts, followed by a collect, as shown below:
def foo(in: Iterator[String]): Iterator[String] =
in.foldLeft((Map.empty[String, Int], "")){ case ((m, prev), word) =>
val count = if (word == prev) m.getOrElse(word, 0) + 1 else 1
(m + (word -> count), word)
}._1.
collect{ case (word, count) if count > 1 => word }.
iterator
foo(Iterator("aaa", "aaa", "bb", "cc", "cc", "bb", "dd")).toList
// res1: List[String] = List("aaa", "cc")
To capture also the repeated word counts and indexes, just index the collection and apply similar tactic for the conditional word count:
def bar(in: Iterator[String]): Map[(String, Int), Int] =
in.zipWithIndex.foldLeft((Map.empty[(String, Int), Int], "", 0)){
case ((m, pWord, pIdx), (word, idx)) =>
val idx1 = if (word == pWord) idx min pIdx else idx
val count = if (word == pWord) m.getOrElse((word, idx1), 0) + 1 else 1
(m + ((word, idx1) -> count), word, idx1)
}._1.
filter{ case ((_, _), count) => count > 1 }
bar(Iterator("aaa", "aaa", "bb", "cc", "cc", "bb", "dd", "cc", "cc", "cc"))
// res2: Map[(String, Int), Int] = Map(("cc", 7) -> 3, ("cc", 3) -> 2, ("aaa", 0) -> 2)
UPDATE:
As per the revised requirement, to minimize memory usage, one approach would be to keep the Map to a minimal size by removing elements of count 1 (which would be the majority if few words are repeated) on-the-fly during the foldLeft traversal. Method baz below is a revised version of bar:
def baz(in: Iterator[String]): Map[(String, Int), Int] =
(in ++ Iterator("")).zipWithIndex.
foldLeft((Map.empty[(String, Int), Int], (("", 0), 0), 0)){
case ((m, pElem, pIdx), (word, idx)) =>
val sameWord = word == pElem._1._1
val idx1 = if (sameWord) idx min pIdx else idx
val count = if (sameWord) m.getOrElse((word, idx1), 0) + 1 else 1
val elem = ((word, idx1), count)
val newMap = m + ((word, idx1) -> count)
if (sameWord) {
(newMap, elem, idx1)
} else
if (pElem._2 == 1)
(newMap - pElem._1, elem, idx1)
else
(newMap, elem, idx1)
}._1.
filter{ case ((word, _), _) => word != "" }
baz(Iterator("aaa", "aaa", "bb", "cc", "cc", "bb", "dd", "cc", "cc", "cc"))
// res3: Map[(String, Int), Int] = Map(("aaa", 0) -> 2, ("cc", 3) -> 2, ("cc", 7) -> 3)
Note that the dummy empty String appended to the input collection is to ensure that the last word gets properly processed as well.
来源：https://stackoverflow.com/questions/59398665/iterator-of-repeated-words-in-a-file