Force Python Scrapy not to encode URL

问题

There are some URLs with [] in it like

http://www.website.com/CN.html?value_ids[]=33&value_ids[]=5007

But when I try scraping this URL with Scrapy, it makes Request to this URL

http://www.website.com/CN.html?value_ids%5B%5D=33&value_ids%5B%5D=5007

How can I force scrapy to not to urlenccode my URLs?

回答1:

When creating a Request object scrapy applies some url encoding methods. To revert these you can utilize a custom middleware and change the url to your needs.

You could use a Downloader Middleware like this:

class MyCustomDownloaderMiddleware(object):

    def process_request(self, request, spider):
        request._url = request.url.replace("%5B", "[", 2)
        request._url = request.url.replace("%5D", "]", 2)

Don't forget to "activate" the middleware in settings.py like so:

DOWNLOADER_MIDDLEWARES = {
    'so.middlewares.MyCustomDownloaderMiddleware': 900,
}

My project is named so and in the folder there is a file middlewares.py. You need to adjust those to your environment.

来源：https://stackoverflow.com/questions/42445087/force-python-scrapy-not-to-encode-url