Streaming in slick/scala

问题

I'm looking at scala/slick streaming, and trying to understand how it works. Here is my test code

    val bigdata = TableQuery[BigData] 
    val x = db.stream(bigdata.result.transactionally.withStatementParameters(fetchSize = 100)).foreach {
      (tuple: (Int, UUID)) =>
        println(tuple._1 + " " + tuple._2)
        Thread.sleep(50)//emulating slow consumer.
    }

    Await.result(x, 100000 seconds)

While the code is running I enabled postgresql query log to understand whats going under the hood. I see a re-query happening every 100 elements

2015-11-06 15:03:24 IST [24379-3] postgres@scala_test LOG: execute fetch from S_2/C_3: select x2."id", x2."data" from "bigdata" x2 2015-11-06 15:03:29 IST [24379-4] postgres@scala_test LOG: execute

fetch from S_2/C_3: select x2."id", x2."data" from "bigdata" x2
2015-11-06 15:03:34 IST [24379-5] postgres@scala_test LOG:  execute fetch from S_2/C_3: select x2."id", x2."data" from "bigdata" x2
2015-11-06 15:03:39 IST [24379-6] postgres@scala_test LOG:  execute fetch from S_2/C_3: select x2."id", x2."data" from "bigdata" x2
2015-11-06 15:03:44 IST [24379-7] postgres@scala_test LOG:  execute fetch from S_2/C_3: select x2."id", x2."data" from "bigdata" x2
2015-11-06 15:03:49 IST [24379-8] postgres@scala_test LOG:  execute fetch from S_2/C_3: select x2."id", x2."data" from "bigdata" x2

However it looks like it was fetching entire dataset. I was expecting a query with offset.

ie SELECT * FROM bigdata  LIMIT 100 OFFSET 500

Looks like everything is queried and send data is send partially.

Then while above streaming is running, I inserted new set of data to same table.

Prior to streaming

SELECT count(*) FROM bigdata -> 500

Then inserted few rows

SELECT count(*) FROM bigdata  -> 700

But the streaming stops at 500. This seems to indicate new data is never fetched and streamed back. Any ideas how streaming works in slick.

来源：https://stackoverflow.com/questions/33565612/streaming-in-slick-scala

标签

scala

streaming

slick

reactive-programming