screen-scraping

Say I am retrieving a list of Urls from a server using Urllib2 library from Python. I noticed that it took about 5 seconds to get one page and it would take a long time to finish all the pages I want to collect. I am thinking out of those 5 seconds. Most of the time was consumed on the server side and I am wondering could I just start using the threading library. Say 5 threads in this case, then the average time could be dramatically increased. Maybe 1 or 2 seconds in each page. (might make the server a bit busy). How could I optimize the number of threads so I could get a legit speed and not

I googled, but I can't find a satisfactory answer. This SO question is related but kinda old as well as the exact opposite of what I am looking for: a way to do screen-scraping using XPath, not CSS selectors. I've used enlive for some basic screen-scraping but sometimes one needs the power of XPath selectors. So here it is: Is there any equivalent to Nokogiri or lxml for clojure (java)? What is the state of the "pure java Nokogiri"? Any way to use the library from clojure? Any better alternatives than this hack ? There are a couple of possibilities here. Several of these require semi-well

How do I scrape an image present on a particular URL using Nokogiri? If there are better options than Nokogiri please suggest. The css image tag is .profilePic img Phrogz If it is just an <img> with a URL: PAGE = "http://site.com/page.html" require 'nokogiri' require 'open-uri' html = Nokogiri.HTML(open(PAGE)) src = html.at('.profilePic img')['src'] File.open("foo.png", "wb") do |f| f.write(open(src).read) end If you need to turn a relative image path into an absolute, see: https://stackoverflow.com/a/4864170/405017 The lazy way is to use mechanize as it will figure out the urls and filenames

I have 9 screen coordinates, each representing one of 9 positions. From the top right, I want that position to start as the 1st position, and the following clockwise coordinates to represent the 2nd, 3rd, 4th and so on, up until the 9th, which would be the top left coordinate. Would anybody here be able to come up with some sort of mathematical means of determining which of the 9 coordinates is in which position? They're all relative to each other, and will always be THAT relative to each other. Example coordinates could be: (x,y) X Y 663 382 543 454 303 454 183 382 418 459 543 209 303 209 653

I have been using open_uri to pull down an ftp path as a data source for some time, but suddenly found that I'm getting nearly continual "530 Sorry, the maximum number of allowed clients (95) are already connected." I am not sure if my code is faulty or if it is someone else who's accessing the server and unfortunately there's no way for me to really seemingly know for sure who's at fault. Essentially I am reading FTP URI's with: def self.read_uri(uri) begin uri = open(uri).read uri == "Error" ? nil : uri rescue OpenURI::HTTPError nil end end I'm guessing that I need to add some additional

I am looking for a free .NET OCR library that will be able to do OCR on a given application window or even a image in memory (I can take a snapshot of the application window myself). I have looked at tessnet2 and MODI but both require an image located on disk. I need to use OCR because the application I am trying to write a script for does some wacky stuff that cannot be read using windows API and I need to scrape data from the screen. I have tested both of tessnet2 and MODI and they both can read the text mostly but because this has to run in an enviroment that will not be able to write to