Doing HTTP request in Scala

Question

I am trying to issue a simple POST request to a webservice which returns some XML in Scala.

It seems that Dispatch is the standard library used for this task, but I cannot find documentation for it. The main site, which I link above, explains at length what is a promise and how to do asynchronous programming, but does not actually document the API. There is a periodic table - which looks a bit scary - but it only seems useful to people who already know what to do and only need a reminder for the cryptic syntax.

It also seems that Scalaz has some facility for HTTP, but I cannot find any documentation for it either.

Answer 1

I use the following: https://github.com/scalaj/scalaj-http.

Here's a simple GET request:

import scalaj.http.Http

Http("http://foo.com/search").param("q", "monkeys").asString

and an example of a POST:

val result = Http("http://example.com/url").postData("""{"id":"12","json":"data"}""")
  .header("Content-Type", "application/json")
  .header("Charset", "UTF-8")
  .option(HttpOptions.readTimeout(10000)).asString

Scalaj HTTP is available through SBT:

libraryDependencies += "org.scalaj" % "scalaj-http_2.11" % "2.3.0"

Answer 2

You could use spray-client. The documentation is lacking (it took me some digging to find out how to make GET requests with query parameters) but it's a great option if you are already using spray. And the documentation is better than dispatch.

We're using it at AI2 over dispatch because the operators are less symbolic and we're already using spray/actors.

import spray.client.pipelining._

val url = "http://youruri.com/yo"
val pipeline: HttpRequest => Future[HttpResponse] = sendReceive

// Post with header and parameters
val responseFuture1: Future[String] = pipeline(Post(Uri(url) withParams ("param" -> paramValue), yourPostData) map (_.entity.asString)

// Post with header
val responseFuture2: Future[String] = pipeline(Post(url, yourPostData)) map (_.entity.asString)

Answer 3

I'm using dispatch: http://dispatch.databinder.net/Dispatch.html

They've just released a new version (0.9.0) with a complete new api that I really like. And it is async.

Example from project page:

import dispatch._
val svc = url("http://api.hostip.info/country.php")
val country = Http(svc OK as.String)

for (c <- country)
  println(c)

edit: This might help you https://github.com/dispatch/reboot/blob/master/core/src/main/scala/requests.scala

Answer 4

Another option is Typesafe's play-ws, which is the Play Framework WS library broken out as a standalone lib:

http://blog.devalias.net/post/89810672067/play-framework-seperated-ws-library

I wouldn't necessarily offer this as the best option, but worth mentioning.

Answer 5

Why not use Apache HttpComponents ? Here's the application FAQ, which covers a wide range of scenarios.

Answer 6

If I can make a shameless plug, I have an API called Bee-Client which is simply a wrapper in Scala for Java's HttpUrlConnection.

Answer 7

I had to do the same to test one end point (in Integration test). So following is the code to fetch response from GET request in Scala. I am making use of scala.io.Source to read from endpoint and ObjectMapper for json to object conversion.

private def buildStockMasterUrl(url:String, stockStatus:Option[String]) = {
      stockStatus match  {
        case Some(stockStatus) => s"$url?stockStatus=${stockStatus}"
        case _ => url
    }
  }

    private def fetchBooksMasterData(stockStatus:Option[String]):  util.ArrayList[BooksMasterData] = {
    val url: String = buildBooksMasterUrl("http://localhost:8090/books/rest/catalogue/booksMasterData",stockStatus)
    val booksMasterJson : String = scala.io.Source.fromURL(url).mkString
    val mapper = new ObjectMapper()
    apper.readValue(booksMasterJson,classOf[util.ArrayList[BooksMasterData]])
}

case class BooksMasterData(id:String,description: String,category: String)

And here is my test method for the same

test("validate booksMasterData resource") {
    val booksMasterData = fetchBooksMasterData(Option(null))
    booksMasterData.size should be (740)
  }

Answer 8

Using my Requests-Scala library:

// Mill
ivy"com.lihaoyi::requests:0.1.8"
// SBT
"com.lihaoyi" %% "requests" % "0.1.8"

This is as simple as

val r = requests.get("https://api.github.com/users/lihaoyi")

r.statusCode
// 200

r.headers("content-type")
// Buffer("application/json; charset=utf-8")

r.text
// {"login":"lihaoyi","id":934140,"node_id":"MDQ6VXNlcjkzNDE0MA==",...

val r = requests.post("http://httpbin.org/post", data = Map("key" -> "value"))

val r = requests.put("http://httpbin.org/put", data = Map("key" -> "value"))

val r = requests.delete("http://httpbin.org/delete")

val r = requests.head("http://httpbin.org/head")

val r = requests.options("http://httpbin.org/get")

Answer 9

Here is a class I was working on. It has both GET and POST requests. GET without parameters - POST with parameters I used it to communicate with StreamSets in order to start a pipeline or check a pipeline status.

it only need the following dependency in the build.sbt file:

libraryDependencies += "org.scalaj" %% "scalaj-http" % "2.3.0"

You can find the documentation here: https://github.com/scalaj/scalaj-http#post-raw-arraybyte-or-string-data-and-get-response-code

import scala.collection.mutable.ArrayBuffer
import scalaj.http.{Http, HttpResponse}

object HttpRequestHandler {

  val userName: String = "admin"
  val password: String = "admin"

  def sendHttpGetRequest(request: String): String = {

    println(" Send Http Get Request (Start) ")

    try {

      val httpResponse: HttpResponse[String] = Http(request).auth(userName,password)
                                                            .asString

      val response = if (httpResponse.code == 200) httpResponse.body
      else{
        println("Bad HTTP response: code = "+httpResponse.code )
        return "ERROR"
      }

      println(" Send Http Get Request (End) ")

      return response

    } catch {
      case e: Exception => println("Error in sending Get request: "+e.getMessage)
        return "ERROR"
    }


  }

  def arrayBufferToJson(params:ArrayBuffer[(String,String)]): String ={

    var jsonString = "{"
    var count: Int = 0
    for(param <- params){
      jsonString+="\""+param._1+"\":\""+param._2+"\""+ ( if(count!=params.length-1) "," else "")
      count+=1
    }
    jsonString+="}"

    return jsonString

  }

  def sendHttpPostRequest(request: String,params: ArrayBuffer[(String,String)]): String = {

    println(" Send Http Post Request (Start) ")

    try {
      val postData : String = arrayBufferToJson(params)
      println("Parameters: "+postData)
      val httpResponse: HttpResponse[String] = Http(request).auth(userName,password)
                                                            .header("X-Requested-By","sdc")
                                                            .header("Content-Type", "application/json;charset=UTF-8")
                                                            .header("X-Stream" , "true")
                                                            .header("Accept", "application/json")
                                                            .postData(postData.getBytes)
                                                            .asString


      val response = if (httpResponse.code == 200) httpResponse.body
      else{
        println("Bad HTTP response: code = "+httpResponse.code )
        "ERROR"
      }

      println(" Send Http Post Request (End) ")

      return response

    } catch {
      case e: Exception => println("Error in sending Post request: " + e.getMessage)
        return "ERROR"
    }
  }

}