I want to crawl a website which supports only post data. I want to send the query params in post data in all the requests. How to achieve this?
alecxe
POST requests can be made using scrapy's Request or FormRequest classes.
Also, consider using start_requests() method instead of start_urls property.
Example:
from scrapy.http import FormRequest
class myspiderSpider(Spider):
name = "myspider"
allowed_domains = ["www.example.com"]
def start_requests(self):
return [ FormRequest("http://www.example.com/login",
formdata={'someparam': 'foo', 'otherparam': 'bar'},
callback=self.parse) ]
Hope that helps.
来源：https://stackoverflow.com/questions/17625053/how-to-send-post-data-in-start-urls-of-the-scrapy-spider