AkkaHTTP: Getting content out of an HttpEntity

Question

I'm working with AkkaHTTP in Scala, trying to interact with an API, but I'm having some trouble getting data with certain API calls. Basically, I have a curl command which works in my terminal for any valid API call to this certain API, but in my current code I only get the actual data for calls whose responses are small enough.

My first attempt was using sys.process._ and the !! method on a string with "curl (url) -X GET -H \"Authorization: Bearer (auth token)\" (which is the command that works in my terminal), but that failed with an error in scala due to issues parsing the header; so after making no headway on that and realizing I wanted to use AkkaHTTP anyway, I abandoned that approach and came to the code below, whose format I borrowed from another stackoverflow post.

implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
implicit val executionContext = system.dispatcher

val responseFuture: Future[HttpResponse] = Http().singleRequest(HttpRequest(
  uri = "(url)").withHeaders(
  RawHeader("Authorization", "Bearer (auth token)")
))


responseFuture
  .onComplete {
    case Success(res) => println(res)
    case Failure(_)   => "failure"
  }

Now, I've tried this on two different API calls; the first, which in my terminal returns a JSON string with 327 characters, returns a blob, where res.entity is HttpEntity.Strict(application/json,(the same JSON blob as in the terminal)). However, while the second call returns in my terminal a JSON blob with 203,413 characters (much bigger than the first one), sending that request in scala returns a blob where res.entity is just HttpEntity.Chunked(application/json). I would presume the payload is split into multiple packets ("Chunked" being a fairly applicable name for that scenario), but I can't seem to figure out how to get the payload information from here, and I'd appreciate some help.

Answer 1

If it's acceptable to load the entire entity into memory, you can call toStrict on the ResponseEntity:

import scala.concurrent.Future
import scala.concurrent.duration._

val entityFut: Future[String] =
  responseFuture.flatMap(_.entity.toStrict(5 seconds).map(_.data.utf8String))

entityFut.onComplete {
  case Success(body) =>
    println(body)
  case Failure(_) =>
    println("failure")
}