How to generate a big data stream on the fly

Question

I have to generate a big file on the fly. Reading to the database and send it to the client. I read some documentation and i did this

val streamContent: Enumerator[Array[Byte]] = Enumerator.outputStream {
        os => 
              // new PrintWriter() read from database and for each record 
              // do some logic and write
              // to outputstream
      }
      Ok.stream(streamContent.andThen(Enumerator.eof)).withHeaders(
              CONTENT_DISPOSITION -> s"attachment; filename=someName.csv"
        )

Im rather new to scala in general only a week so don't guide for my reputation.

My questions are :

1) Is this the best way? I found this if i have a big file, this will load in memory, and also don't know what is the chunk size in this case, if it will send for each write() is not to convenient.

2) I found this method Enumerator.fromStream(data : InputStream, chunkedSize : int) a little better cause it has a chunk-size, but i don't have an inputStream cause im creating the file on the fly.

Answer 1

There's a note in the docs for Enumerator.outputStream:

Not [sic!] that calls to write will not block, so if the iteratee that is being fed to is slow to consume the input, the OutputStream will not push back. This means it should not be used with large streams since there is a risk of running out of memory.

If this can happen depends on your situation. If you can and will generate Gigabytes in seconds, you should probably try something different. I'm not exactly sure what, but I'd start at Enumerator.generateM(). For many cases though, your method is perfectly fine. Have a look at this example by Gaëtan Renaudeau for serving a Zip file that's generated on the fly in the same way you're using it:

val enumerator = Enumerator.outputStream { os =>
  val zip = new ZipOutputStream(os);
  Range(0, 100).map { i =>
    zip.putNextEntry(new ZipEntry("test-zip/README-"+i+".txt"))
    zip.write("Here are 100000 random numbers:\n".map(_.toByte).toArray)
    // Let's do 100 writes of 1'000 numbers
    Range(0, 100).map { j =>
      zip.write((Range(0, 1000).map(_=>r.nextLong).map(_.toString).mkString("\n")).map(_.toByte).toArray);
    }
    zip.closeEntry()
  }
  zip.close()
}
Ok.stream(enumerator >>> Enumerator.eof).withHeaders(
  "Content-Type"->"application/zip", 
  "Content-Disposition"->"attachment; filename=test.zip"
)

Please keep in mind that Ok.stream has been replaced by Ok.chunked in newer versions of Play, in case you want to upgrade.

As for the chunk size, you can always use Enumeratee.grouped to gather a bunch of values and send them as one chunk.

val grouper = Enumeratee.grouped(  
  Traversable.take[Array[Double]](100) &>> Iteratee.consume()  
)

Then you'd do something like

Ok.stream(enumerator &> grouper >>> Enumerator.eof)