Scrapy and Selenium submit form that renders dynamically

问题

I'm using selenium to first load the form which being generated via ajax.

Now I'm having troubles passing the selenium response to scrapy FormReuqest method

to send the form data values.

The form has jquery validation before user can submit it, does it make it harder to submit using scrapy?

any help is appreciated.

thanks

回答1:

You don't need neither Scrapy, nor selenium here. Just make an underlying POST request and parse the json response. Example using requests:

import json
import requests


URL = 'http://calculator.shipito.com/en/rates'
data = {"location": "10", "country": "GB", "city": "London",
        "postalcode": "123456", 
        "packages": [{"dimensions_units": "in", "weight_units": "lbs", "dimension": {"width": "9", "height": "20.4", "length": "17"},
        "weight": "9", "value": "170"}]}

response = requests.post(URL, data=json.dumps(data))
print response.json()

Prints:

[
    {u'infoUrl': u'http://www.shipito.com/postage#tnt', u'isPromoted': False, u'popularity': 1, u'name': u'TNT Economy', u'deliveryTime': 6, u'insurable': True, u'usesDimWeight': True, u'shippingRate': 68.38, u'separateShippingRate': None, u'bonusShippingRate': 68.38, u'deliveryTimeInfo': u'4-6 business days', u'mps': False, u'insuranceRate': 0, u'logoImg': u'tnt.png'}, 
    ...
]

来源：https://stackoverflow.com/questions/24864832/scrapy-and-selenium-submit-form-that-renders-dynamically