rcurl | 易学教程

How to login and then download a file from aspx web pages with R

阅读更多关于 How to login and then download a file from aspx web pages with R

I'm trying to automate the download of the Panel Study of Income Dynamics files available on this web page using R. Clicking on any of those files takes the user through to this login/authentication page . After authentication, it's easy to download the files with your web browser. Unfortunately, the httr code below does not appear to be maintaining the authentication. I have tried inspecting the Headers in Chrome for the Login.aspx page ( as described here ), but it doesn't appear to maintain the authentication even when I believe I'm passing in all the correct values. I don't care if it's

Oauth with Twitter Streaming API in R (using RCurl)

阅读更多关于 Oauth with Twitter Streaming API in R (using RCurl)

问题 I would like to connect to Twitter's Streaming API using RCurl in R, and also be able to filter keywords. However, new restrictions on authorization in Twitter API v1.1 is making using RCurl difficult. Before, code could go something like this taken from this page: getURL("https://stream.twitter.com/1/statuses/filter.json", userpwd="Username:Password", cainfo = "cacert.pem", write=my.function, postfields="track=bruins") But now, Twitter's new API is making users authorize with OAuth. I have a

How can I screenshot a website using R?

阅读更多关于 How can I screenshot a website using R?

问题 So I'm not 100% sure this is possible, but I found a good solution in Ruby and in python, so I was wondering if something similar might work in R. Basically, given a URL, I want to render that URL, take a screenshot of the rendering as a .png, and save the screenshot to a specified folder. I'd like to do all of this on a headless linux server. Is my best solution here going to be running system calls to a tool like CutyCapt, or does there exist an R-based toolset that will help me solve this

using Rcurl with HTTPs

阅读更多关于 using Rcurl with HTTPs

问题 I tried the following code in R on windows: library(RCurl) postForm("https://www.google.com/accounts/ClientLogin/", "email" = "me@gmail.com", "Passwd" = "abcd", "service" = "finance", "source" = "Test-1" ) but go the following error: Error in postForm() SL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed How do I setup RCurl to allow use of HTTPs? 回答1: You need to install a SSL library. For windows

SOAP request in R

阅读更多关于 SOAP request in R

问题 Does anyone know how to formulate following SOAP request with R? POST /API/v201010/AdvertiserService.asmx HTTP/1.1 Host: advertising.criteo.com Content-Type: text/xml; charset=utf-8 Content-Length: length SOAPAction: "https://advertising.criteo.com/API/v201010/clientLogin" <?xml version="1.0" encoding="utf-8"?> <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"> <soap:Body>

extract data from raw html in R

阅读更多关于 extract data from raw html in R

问题 I am trying to extract the values of all the values in all tabs from this page. http://www.imd.gov.in/section/hydro/dynamic/rfmaps/weekrain.htm I first tried downloading as excel. But that was not possible. I am just able to download it as text file. If I try reading directly from webpage I get the raw html page. I am stuck as how to extract these values. Please find the code which I tried till now. library(RCurl) require(XML) url = "http://www.imd.gov.in/section/hydro/dynamic/rfmaps/weekrain

How to get google search results

阅读更多关于 How to get google search results

问题 I used the following code: library(XML) library(RCurl) getGoogleURL <- function(search.term, domain = '.co.uk', quotes=TRUE) { search.term <- gsub(' ', '%20', search.term) if(quotes) search.term <- paste('%22', search.term, '%22', sep='') getGoogleURL <- paste('http://www.google', domain, '/search?q=', search.term, sep='') } getGoogleLinks <- function(google.url) { doc <- getURL(google.url, httpheader = c("User-Agent" = "R(2.10.0)")) html <- htmlTreeParse(doc, useInternalNodes = TRUE, error

Scrape password-protected website in R

阅读更多关于 Scrape password-protected website in R

I'm trying to scrape data from a password-protected website in R. Reading around, it seems that the httr and RCurl packages are the best options for scraping with password authentication (I've also looked into the XML package). The website I'm trying to scrape is below (you need a free account in order to access the full page): http://subscribers.footballguys.com/myfbg/myviewprojections.php?projector=2 Here are my two attempts (replacing "username" with my username and "password" with my password): #This returns "Status: 200" without the data from the page: library(httr) GET("http:/

R - How to make a click on webpage using rvest or rcurl

阅读更多关于 R - How to make a click on webpage using rvest or rcurl

问题 I want to download data from this webpage The data can be easily scraped with rvest . The code maybe like this : library(rvest) library(pipeR) url <- "http://www.tradingeconomics.com/" css <- "#ctl00_ContentPlaceHolder1_defaultUC1_CurrencyMatrixAllCountries1_GridView1" data <- url %>>% html() %>>% html_nodes(css) %>>% html_table() But there is a problem for webpages like this. There is a + button to show the data of all the countries, but the default is just data of 50 countries. So if I use

R: extracting “clean” UTF-8 text from a web page scraped with RCurl

阅读更多关于 R: extracting “clean” UTF-8 text from a web page scraped with RCurl

问题 Using R, I am trying to scrape a web page save the text, which is in Japanese, to a file. Ultimately this needs to be scaled to tackle hundreds of pages on a daily basis. I already have a workable solution in Perl, but I am trying to migrate the script to R to reduce the cognitive load of switching between multiple languages. So far I am not succeeding. Related questions seem to be this one on saving csv files and this one on writing Hebrew to a HTML file. However, I haven't been successful