scrapy-pipeline

问题 I have created an item pipeline as an answer to this question. It is supposed to create a new file for every page according to the page_no value set in the item. This works mostly fine. The problem is with the last csv file generated by the pipeline/item exporter, page-10.csv . The last 10 values are not exported, so the file stays empty. What could be the reason for this behaviour? pipelines.py from scrapy.exporters import CsvItemExporter class PerFilenameExportPipeline: """Distribute items

问题 I'm trying to build a small app for a university project with Scrapy. The spider is scraping the items, but my pipeline is not inserting data into mysql database. In order to test whether the pipeline is not working or the pymysl implementation is not working I wrote a test script: Code Start #!/usr/bin/python3 import pymysql str1 = "hey" str2 = "there" str3 = "little" str4 = "script" db = pymysql.connect("localhost","root","**********","stromtarife" ) cursor = db.cursor() cursor.execute(

问题 I'm trying to create a custom pipeline for a Scrapy project that outputs the collected items to CSV files. In order to keep each file's size down I want to set a maximum number of rows that each file can have. Once the line limit has been reached in the current file a new file is created to continue outputting the items. Luckily, I found a question where someone was looking to do the same thing. And there's an answer to that question that shows an example implementation. I implemented the

问题 I'm trying to create a custom pipeline for a Scrapy project that outputs the collected items to CSV files. In order to keep each file's size down I want to set a maximum number of rows that each file can have. Once the line limit has been reached in the current file a new file is created to continue outputting the items. Luckily, I found a question where someone was looking to do the same thing. And there's an answer to that question that shows an example implementation. I implemented the

问题 I've a spider which fetches both the data and images. I want to rename the images with the respective 'title' which i'm fetching. Following is my code: spider1.py from imageToFileSystemCheck.items import ImagetofilesystemcheckItem import scrapy class TestSpider(scrapy.Spider): name = 'imagecheck' def start_requests(self): searchterms=['keyword1','keyword2',] for item in searchterms: yield scrapy.Request('http://www.example.com/s?=%s' % item,callback=self.parse, meta={'item': item}) def parse

问题 My requirement is to dump scraped items to two different csv files. I'm able to scrape the data but CSV file is empty. Could anyone please help in this regard. Below is the code for the pipeline.py file and console logs: Code for pipeline.py : # -*- coding: utf-8 -*- # Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html from scrapy.exporters import CsvItemExporter from scrapy

问题 My requirement is to dump scraped items to two different csv files. I'm able to scrape the data but CSV file is empty. Could anyone please help in this regard. Below is the code for the pipeline.py file and console logs: Code for pipeline.py : # -*- coding: utf-8 -*- # Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html from scrapy.exporters import CsvItemExporter from scrapy

问题 My requirement is to dump scraped items to two different csv files. I'm able to scrape the data but CSV file is empty. Could anyone please help in this regard. Below is the code for the pipeline.py file and console logs: Code for pipeline.py : # -*- coding: utf-8 -*- # Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html from scrapy.exporters import CsvItemExporter from scrapy

问题 My requirement is to dump scraped items to two different csv files. I'm able to scrape the data but CSV file is empty. Could anyone please help in this regard. Below is the code for the pipeline.py file and console logs: Code for pipeline.py : # -*- coding: utf-8 -*- # Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html from scrapy.exporters import CsvItemExporter from scrapy

问题 I am importing the Scrapy item keys from items.py , into pipelines.py . The problem is that the order of the imported items are different from how they were defined in the items.py file. My items.py file: class NewAdsItem(Item): AdId = Field() DateR = Field() AdURL = Field() In my pipelines.py : from adbot.items import NewAdsItem ... def open_spider(self, spider): self.ikeys = NewAdsItem.fields.keys() print("Keys in pipelines: \t%s" % ",".join(self.ikeys) ) #self.createDbTable(ikeys) The