scrapy-spider

I have two CrawlerProcesses, each is calling different spider. I want to pass custom settings to one of these processes to save the output of the spider to csv, I thought I could do this: storage_settings = {'FEED_FORMAT': 'csv', 'FEED_URI': 'foo.csv'} process = CrawlerProcess(get_project_settings()) process.crawl('ABC', crawl_links=main_links, custom_settings=storage_settings ) process.start() and in my spider I read them as an argument: def __init__(self, crawl_links=None, allowed_domains=None, customom_settings=None, *args, **kwargs): self.start_urls = crawl_links self.allowed_domains =

I scrap successfully data for a single account. I want to scrap multiple accounts on a single website, multiple accounts needs multiple logins, I want a way how to manage login/logout ? dangra you can scrape multiples accounts in parallel using multiple cookiejars per account session, see "cookiejar" request meta key at http://doc.scrapy.org/en/latest/topics/downloader-middleware.html?highlight=cookiejar#std:reqmeta-cookiejar To clarify: suppose we have an array of accounts in settings.py : MY_ACCOUNTS = [ {'login': 'my_login_1', 'pwd': 'my_pwd_1'}, {'login': 'my_login_2', 'pwd': 'my_pwd_2'},

I use scrapy to crawl 1000 urls and store scraped item in a mongodb. I'd to know how many items have been found for each url. From scrapy stats I can see 'item_scraped_count': 3500 However, I need this count for each start_url separately. There is also referer field for each item that I might use to count each url items manually: 2016-05-24 15:15:10 [scrapy] DEBUG: Crawled (200) <GET https://www.youtube.com/watch?v=6w-_ucPV674> (referer: https://www.youtube.com/results?q=billys&sp=EgQIAhAB) But I wonder if there is a built-in support from scrapy. eLRuLL challenge accepted! there isn't

I just started to learn scrapy. So I followed the scrapy documentation . I just written the first spider mentioned in that site. import scrapy class DmozSpider(scrapy.Spider): name = "dmoz" allowed_domains = ["dmoz.org"] start_urls = [ "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/", "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/" ] def parse(self, response): filename = response.url.split("/")[-2] with open(filename, 'wb') as f: f.write(response.body) Upon running this scrapy crawl dmoz command on project's root directory, it shows the below error.

I've written a script in python's scrapy to make a proxied requests using either of the newly generated proxies by get_proxies() method. I used requests module to fetch the proxies in order to reuse them in the script. However, the problem is the proxy my script chooses to use may not be the good one always so sometimes it doesn't fetch valid response. How can I let my script keep trying with different proxies until there is a valid response? My script so far: import scrapy import random import requests from itertools import cycle from bs4 import BeautifulSoup from scrapy.http.request import

I'm trying to submit a dynamically generated user login form using Scrapy and then parse the HTML on the page that corresponds to a successful login. I was wondering how I could do that with Scrapy or a combination of Scrapy and Selenium. Selenium makes it possible to find the element on the DOM, but I was wondering if it would be possible to "give control back" to Scrapy after getting the full HTML in order to allow it to carry out the form submission and save the necessary cookies, session data etc. in order to scrape the page. Basically, the only reason I thought Selenium was necessary was

After installing all dependencies for scrapy on windows 32bit. I've tried to build an executable from my scrapy spider. Spider script "runspider.py" works ok when running as "python runspider.py" Building executable "pyinstaller --onefile runspider.py": C:\Users\username\Documents\scrapyexe>pyinstaller --onefile runspider.py 19 INFO: wrote C:\Users\username\Documents\scrapyexe\runspider.spec 49 INFO: Testing for ability to set icons, version resources... 59 INFO: ... resource update available 59 INFO: UPX is not available. 89 INFO: Processing hook hook-os 279 INFO: Processing hook hook-time