Scrapy FormRequest sending JSON

Question

I am trying to create a FormRequest that can send content-type:application/json.

Here is what I try:

yield FormRequest(\"abc.someurl.com\", formdata=         


        

Answer 1

You could just use the below snippet for getting response:

res =requests.post(url, json=data) print(res.text)

Answer 2

from scrapy.http import JsonRequest

class TestSpider(scrapy.Spider):
    def start_requests(self):
        data = {
            'name1': 'value1',
            'name2': 'value2',
        }
        yield JsonRequest(url='http://www.example.com/post/action', data=data)

see more: docs.scrapy.org#jsonrequest

Answer 3

FormRequest is for simulating an HTML form (e.g. application/x-www-form-urlencoded). It sounds like you are simply wanting to POST data with your Request. Since you mention a content type of 'application/json' you probably want to do something like this:

request = Request( url, method='POST', 
                   body=json.dumps(my_data), 
                   headers={'Content-Type':'application/json'} )