问题
I am trying to extract data from airbnb.com. But whenever I tried to access that website with .com in its domain, it's redirecting to a domain having .ca.
Here is a code snippet which I think would describe my issue
In [46]: fetch(url)
2021-02-05 09:17:36 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.airbnb.ca/s/nova/homes?tab_id=home_tab&refinement_paths%5B%5D=%2Fhomes&date_picker_type=calendar&source=structured_search_input_header&search_type=search_query&locale=en&_set_bev_on_new_domain=1612519553_NTk3ZTMyZGU4NzI2> from <GET https://www.airbnb.com/s/nova/homes?tab_id=home_tab&refinement_paths[]=%2Fhomes&date_picker_type=calendar&source=structured_search_input_header&search_type=search_query>
2021-02-05 09:17:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.airbnb.ca/s/nova/homes?tab_id=home_tab&refinement_paths%5B%5D=%2Fhomes&date_picker_type=calendar&source=structured_search_input_header&search_type=search_query&locale=en&_set_bev_on_new_domain=1612519553_NTk3ZTMyZGU4NzI2> (referer: None)
Actually, airbnb loads prices for each listing with the domain having .com. I tried various solution like don_redirect etc. but haven't received required response.
来源:https://stackoverflow.com/questions/66064965/stop-scrapy-from-redirecting-to-country-specific-domain