scrape

I'm trying to scrape web pages in a Ruby script that I'm working on. The purpose of the project is to show which ETFs and stock mutual funds are most compatible with the value investing philosophy. Some examples of pages I'd like to scrape are: http://finance.yahoo.com/q/pr?s=SPY+Profile http://finance.yahoo.com/q/hl?s=SPY+Holdings http://www.marketwatch.com/tools/mutual-fund/list/V What web scraping tools do you recommend for Ruby, and why? Keep in mind that there are thousands of stock funds out there, so any tool I use has to be reasonably quick. I am new to Ruby, but I have experience

In the browser, navigating to this URL initiates a 302 (moved temporarily) request which in turn downloads a file. http://www.targetsite.com/target.php/?event=download&task_id=123 When I view what is actually happening via Chrome network tools I see that the redirect is going to a dynamically generated path that cancels itself immediately after download. In other words, even if I know that full path I will not have time to manually call it. So, how in using the command line can I mimic the browser actions? I tried curl --cookies bin/cookies.txt -O -L " http://www.targetsite.com/target.php/

So I am fairly new to web scraping. There is this site that has a table on it, the values of the table are controlled by Javascript. The values will determine the address of future values that my browser is told to request from the Javascript. These new pages have JSON responses that the script updates the table with in my browser. So I wanted to build a class with a mechanize method that takes in an url and spits out the body response, the first time a HTML, afterwards, the body response will be JSON, for remaining iterations. I have something that works but I want to know if I am doing it

I am trying to scrape a website and I am getting a 403 Forbidden error no matter what I try: wget CURL (command line and PHP) Perl WWW::Mechanize PhantomJS I tried all of the above with and without proxies, changing user-agent, and adding a referrer header. I even copied the request header from my Chrome browser and tried sending with my request using PHP Curl and I am still getting a 403 Forbidden error. Any input or suggestions on what is triggering the website to block the request and how to bypass? PHP CURL Example: $url ='https://www.vitacost.com/productResults.aspx?allCategories=true&N

The webpage is something like this: <h2>section1</h2> <p>article</p> <p>article</p> <p>article</p> <h2>section2</h2> <p>article</p> <p>article</p> <p>article</p> How can I find each section with articles within them? That is, after finding h2, find nextsiblings until the next h2. If the webpage were like: (which is normally the case) <div> <h2>section1</h2> <p>article</p> <p>article</p> <p>article</p> </div> <div> <h2>section2</h2> <p>article</p> <p>article</p> <p>article</p> </div> I can write codes like: for section in soup.findAll('div'): ... for post in section.findAll('p') But what should