scrapy-spider

I am following this tutorial to use splash to help with scraping webpages.I installed Docker toolbox and did these two steps: $ docker pull scrapinghub/splash $ docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash I think it is running correctly, based on the prompted message in Docker window, which looks like this: However, when I open the `localhost:8050' in a web browser, it says the localhost is not working. What might have gone wrong in this case? Thanks! VonC You have mapped the port to your docker host (the VM), but you have not port-forwarded that same port to your

I am trying to scrape using scrape framework. Some requests are redirected but the callback function set in the start_requests is not called for these redirected url requests but works fine for the non-redirected ones. I have the following code in the start_requests function: for user in users: yield scrapy.Request(url=userBaseUrl+str(user['userId']),cookies=cookies,headers=headers,dont_filter=True,callback=self.parse_p) But this self.parse_p is called only for the Non-302 requests. I guess you get a callback for the final page (after the redirect). Redirects are been taken care by the

I want to scrape contents from multiple tables in a webpage and the HTML code goes like this : <div class="fixtures-table full-table-medium" id="fixtures-data"> <h2 class="table-header"> Date 1 </h2> <table class="table-stats"> <tbody> <tr class='preview' id='match-row-EFBO755307'> <td class='details'> <p> <span class='team-home teams'> <a href='random_team'>team 1</a> </span> <span class='team-away teams'> <a href='random_team'>team 2</a> </span> </p> </td> </tr> <tr class='preview' id='match-row-EFBO755307'> <td class='match-details'> <p> <span class='team-home teams'> <a href='random_team'

We have a pretty much standard Scrapy project (Scrapy 0.24). I'd like to catch specific HTTP response codes, such as 200, 500, 502, 503, 504 etc. Something like that: class Spider(...): def parse(...): processes HTTP 200 def parse_500(...): processes HTTP 500 errors def parse_502(...): processes HTTP 502 errors ... How can we do that? alecxe By default, Scrapy only handles responses with status codes 200 - 300 . Let Scrapy handle 500 and 502 : class Spider(...): handle_httpstatus_list = [500, 502] Then, in the parse() callback, check response.status : def parse(response): if response.status ==