POST request using RCurl

Question

As a way of exploring how to make a package in R for the Denver RUG, I decided that it would be a fun little project to write an R wrapper around the datasciencetoolkit API. The basic R tools come from the RCurl package as you might imagine. I am stuck on a seemingly simple problem and I'm hoping that somebody in this forum might be able to point me in the right direction. The basic problem is that I can't seem to use postForm() to pass an un-keyed string as part of the data option in curl, i.e. curl -d "string" "address_to_api".

For example, from the command line I might do

$ curl -d "Tim O'Reilly, Archbishop Huxley" "http://www.datasciencetoolkit.org/text2people"

with success. However, it seems that postForm() requires an explicit key when passing additional arguments into the POST request. I've looked through the datasciencetoolkit code and developer docs for a possible key, but can't seem to find anything.

As an aside, it's pretty straightforward to pass inputs via a GET request to other parts of the DSTK API. For example,

ip2coordinates <- function(ip) {
  api <- "http://www.datasciencetoolkit.org/ip2coordinates/"
  result <- getURL(paste(api, URLencode(ip), sep=""))
  names(result) <- "ip"
  return(result)
}
ip2coordinates('67.169.73.113')

will produce the desired results.

To be clear, I've read through the RCurl docs on DTL's omegahat site, the RCurl docs with the package, and the curl man page. However, I'm missing something fundamental with respect to curl (or perhaps .opts() in the postForm() function) and I can't seem to get it.

In python, I could basically make a 'raw' POST request using httplib.HTTPConnection -- is something like that available in R? I've looked at the simplePostToHost function in the httpRequest package as well and it just seemed to lock my R session (it seems to require a key as well).

FWIW, I'm using R 2.13.0 on Mac 10.6.7.

Any help is much appreciated. All of the code will soon be available on github if you're interested in playing around with the data science toolkit.

Cheers.

Answer 1

With httr, this is just:

library(httr)
r <- POST("http://www.datasciencetoolkit.org/text2people", 
  body = "Tim O'Reilly, Archbishop Huxley")
stop_for_status(r)
content(r, "parsed", "application/json")

Answer 2

Generally, in those cases where you're trying to POST something that isn't keyed, you can just assign a dummy key to that value. For example:

> postForm("http://www.datasciencetoolkit.org/text2people", a="Archbishop Huxley")
[1] "[{\"gender\":\"u\",\"first_name\":\"\",\"title\":\"archbishop\",\"surnames\":\"Huxley\",\"start_index\":44,\"end_index\":61,\"matched_string\":\"Archbishop Huxley\"},{\"gender\":\"u\",\"first_name\":\"\",\"title\":\"archbishop\",\"surnames\":\"Huxley\",\"start_index\":88,\"end_index\":105,\"matched_string\":\"Archbishop Huxley\"}]"
attr(,"Content-Type")
                charset 
"text/html"     "utf-8" 

Would work the same if I'd used b="Archbishop Huxley", etc.

Enjoy RCurl - it's probably my favorite R package. If you get adventurous, upgrading to ~ libcurl 7.21 exposes some new methods via curl (including SMTP, etc.).

Answer 3

From Duncan Temple Lang on the R-help list:

postForm() is using a different style (or specifically Content-Type) of submitting the form than the curl -d command. Switching the style = 'POST' uses the same type, but at a quick guess, the parameter name 'a' is causing confusion and the result is the empty JSON array - "[]".

A quick workaround is to use curlPerform() directly rather than postForm()

r = dynCurlReader()
curlPerform(postfields = 'Archbishop Huxley', url = 'http://www.datasciencetoolkit.org/text2people', verbose = TRUE,
             post = 1L, writefunction = r$update)
r$value()

This yields

[1]
"[{\"gender\":\"u\",\"first_name\":\"\",\"title\":\"archbishop\",\"surnames\":\"Huxley\",\"start_index\":0,\"end_index\":17,\"matched_string\":\"Archbishop
Huxley\"}]"

and you can use fromJSON() to transform it into data in R.

Answer 4

I just wanted to point out that there must be an issue with passing a raw string via the postForm function. For example, if I use curl from the command line, I get the following:

    $ curl -d "Archbishop Huxley" "http://www.datasciencetoolkit.org/text2people
[{"gender":"u","first_name":"","title":"archbishop","surnames":"Huxley","start_index":0,"end_index":17,"matched_string":"Archbishop Huxley"}]

and in R I get

> api <- "http://www.datasciencetoolkit.org/text2people"
> postForm(api, a="Archbishop Huxley")
[1] "[{\"gender\":\"u\",\"first_name\":\"\",\"title\":\"archbishop\",\"surnames\":\"Huxley\",\"start_index\":44,\"end_index\":61,\"matched_string\":\"Archbishop Huxley\"},{\"gender\":\"u\",\"first_name\":\"\",\"title\":\"archbishop\",\"surnames\":\"Huxley\",\"start_index\":88,\"end_index\":105,\"matched_string\":\"Archbishop Huxley\"}]"
attr(,"Content-Type")
                charset 
"text/html"     "utf-8" 

Note that it returns two elements in the JSON string and neither one matches on the start_index or end_index. Is this a problem with encoding or something?

Answer 5

The simplePostToHost function in the httpRequest package might do what you are looking for here.