Using R to connect to a sharepoint list

Question

Has anyone been able to import a SharePoint list in R as a dataframe?

I have two separate data sources, one from a SharePoint list and the other from a DB that I wish to run an analysis on. I am able to connect to the DB without any problem but can't seem to find anything to connect to a SharePoint list.

The SharePoint server is 2007

Answer 1

I've been working on reading SharePoint 2010 lists using R for a little while now. Basically, I use the SharePoint web service to return the results from the list, then use xmlToDataFrame to convert to a dataframe.

URL <- "http://yoursharepointserver/_vti_bin/ListData.svc/yourlist"    
data = xmlParse(readLines(URL))

## get the individual list items    
items = getNodeSet(data, "//m:properties")

## convert to a data frame
df = xmlToDataFrame(items, stringsAsFactors = FALSE)

Since I'm using the web service I can filter the list before I return the results, which is really helpful in overcoming the limitations of the SharePoint web service. The following link is quite helpful... http://www.dotnetmafia.com/blogs/dotnettipoftheday/archive/2010/01/21/introduction-to-querying-lists-with-rest-and-listdata-svc-in-sharepoint-2010.aspx

Answer 2

Lee Mendoza's answer may well work if ListData.svc is running and/or you have administrative access to the SharePoint server.

If both of those aren't true: the following might work. At least it does for me on SharePoint 2010. If there's a better way of doing it when ListData.svc isn't present, I'd love to hear it.

 library(RCurl)
 library(XML)
 library(data.table)
 URL <- "http://<site>/_vti_bin/owssvr.dll?Cmd=Display&Query=*&XMLDATA=TRUE&List={GUID_OF_LIST}"
 rawData <- getURL(URL, userpwd = "username:password")
 # in real life  prompt for user credentials, don't put in script
 xmlData <- xmlParse (rawData, options=HUGE, useInternalNodes=TRUE)
 dataList <- xmlToList(xmlRoot(xmlData)[["data"]])
 # check the system return, on my SP2010 server the data block is 
 # named rs:data so this works
 dataMatrix <- do.call(rbind,dataList)
 finalDataTable <- data.table(dataMatrix)

Answer 3

The answer above works for lists which are <= 1000 rows only. Using "$Top" and "$Skip" in the URL, you can use the function below which iterates multiple times and imports all the data from the list regardless of the size. (This may not be the most clean way to write it, but it works!)

sp_import <- function(ListName) {

        urlstring <- "http://yoursharepointserver/_vti_bin/ListData.svc/yourlist" 
        data <- xmlParse(readLines(paste(urlstring, ListName, sep = ""), warn = FALSE))
        items <- getNodeSet(data, "//m:properties")
        df <- xmlToDataFrame(items, stringsAsFactors = FALSE)
        iterate <- nrow(df)
        skip <- 1

        while (nrow(df) == 1000 * skip) {
            data <- xmlParse(readLines(paste(urlstring, ListName, "?$top=1000&$skip=", iterate, sep = ""), warn = FALSE))
            items <- getNodeSet(data, "//m:properties")
            df <- rbind(df, xmlToDataFrame(items, stringsAsFactors = FALSE))
            iterate <- nrow(df)
            skip <- skip + 1
        }
        return(df)
}