scrapy-spider

I've been trying to scrapy the following Website but with the currency changed to 'SAR' from the upper left settings form , i tried sending a scrapy request like this: r = Request(url='https://www.mooda.com/en/', cookies=[{'name': 'currency', 'value': 'SAR', 'domain': '.www.mooda.com', 'path': '/'}, {'name':'country','value':'SA','domain': '.www.mooda.com','path':'/'}],dont_filter=True) and i still get the price as EG In [10]: response.css('.price').xpath('text()').extract() Out[10]: [u'1,957 EG\xa3', u'3,736 EG\xa3', u'2,802 EG\xa3', u'10,380 EG\xa3', u'1,823 EG\xa3'] i have also tried to

As you see, the following json has number of followers as well as number of comments but how can I access the data within each comment as well as ID of followers so I could crawl into them? { "logging_page_id": "profilePage_20327023", "user": { "biography": null, "blocked_by_viewer": false, "connected_fb_page": null, "country_block": false, "external_url": null, "external_url_linkshimmed": null, "followed_by": { "count": 2585 }, "followed_by_viewer": false, "follows": { "count": 561 }, "follows_viewer": false, "full_name": "LeAnne Barengo", "has_blocked_viewer": false, "has_requested_viewer":

I'm new to Scrapy and I'm really just lost on how i can return multiple items in one block. Basically, I'm getting one HTML tag which has a quote that contains nested tags of text, author name, and some tags about that quote. The code here only returns one quote and that's it. It doesnt use the loop to return the rest. I've been searching the web for hours and I'm just hopeless I don't get it. Here's my code so far: Spider.py import scrapy from scrapy.loader import ItemLoader from first_spider.items import FirstSpiderItem class QuotesSpider(scrapy.Spider): name = 'quotes' allowed_domains = [

Help! Reading the following scrapy code and the result of crawler. I want to crawl some data from http://china.fathom.info/data/data.json , and only Scrapy is allowed. But I don't know how to control the order of yield. I look forward to process all parse_member request in the loop and then return the group_item , but seems yield item is always executed before yield request. start_urls = [ "http://china.fathom.info/data/data.json" ] def parse(self, response): groups = json.loads(response.body)['group_members'] for i in groups: group_item = GroupItem() group_item['name'] = groups[i]['name']

i am using Splash 2.0.2 + Scrapy 1.0.5 + Scrapyjs 0.1.1 and im still not able to render javascript with a click. Here is an example url https://olx.pt/anuncio/loja-nova-com-250m2-garagem-em-box-fechada-para-arrumos-IDyTzAT.html#c49d3d94cf I am still getting the page without the phone number rendered: class OlxSpider(scrapy.Spider): name = "olx" rotate_user_agent = True allowed_domains = ["olx.pt"] start_urls = [ "https://olx.pt/imoveis/" ] def parse(self, response): script = """ function main(splash) splash:go(splash.args.url) splash:runjs('document.getElementById("contact_methods")

I've a list of tuples in the form (id,url) I need to crawl a product from a list of urls, and when those products are crawled i need to store them in database under their id. problem is i can't understand how to pass id to parse function so that i can store crawled item under their id. Initialize start urls in start_requests() and pass id in meta : class MySpider(Spider): mapping = [(1, 'my_url1'), (2, 'my_url2')] ... def start_requests(self): for id, url in self.mapping: yield Request(url, callback=self.parse_page, meta={'id': id}) def parse_page(self, response): id = response.meta['id'] 来源：