How to scrape a website with the socksify gem (proxy)

ぃ、小莉子 提交于 2019-12-05 06:45:52

问题


I am reading through the documentation of the socksify gem on Rubyforge. I have installed the gem successfully, and I have run this documented code with success to test that my local implementation can replicate it:

require 'socksify/http'
uri = URI.parse('http://rubyforge.org/')
Net::HTTP.SOCKSProxy('127.0.0.1', 9050).start(uri.host, uri.port) do |http|
  http.get(uri.path)
end
# => #<Net::HTTPOK 200 OK readbody=true>

But how do I e.g. scrape 'http://google.com/', and get the html content? I wish to parse it with e.g. Nokogiri like this:

Nokogiri::HTML(open("http://google.com/))

回答1:


 require 'socksify/http'
 http = Net::HTTP::SOCKSProxy(addr, port)
 html = http.get(URI('http://google.de'))
 html_doc = Nokogiri::HTML(html)


来源:https://stackoverflow.com/questions/21752838/how-to-scrape-a-website-with-the-socksify-gem-proxy

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!