Stop scrapy from redirecting to country specific domain

大憨熊 提交于 2021-02-08 09:00:46

问题


I am trying to extract data from airbnb.com. But whenever I tried to access that website with .com in its domain, it's redirecting to a domain having .ca.

Here is a code snippet which I think would describe my issue

In [46]: fetch(url)                      
2021-02-05 09:17:36 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.airbnb.ca/s/nova/homes?tab_id=home_tab&refinement_paths%5B%5D=%2Fhomes&date_picker_type=calendar&source=structured_search_input_header&search_type=search_query&locale=en&_set_bev_on_new_domain=1612519553_NTk3ZTMyZGU4NzI2> from <GET https://www.airbnb.com/s/nova/homes?tab_id=home_tab&refinement_paths[]=%2Fhomes&date_picker_type=calendar&source=structured_search_input_header&search_type=search_query>
                                                                                    

2021-02-05 09:17:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.airbnb.ca/s/nova/homes?tab_id=home_tab&refinement_paths%5B%5D=%2Fhomes&date_picker_type=calendar&source=structured_search_input_header&search_type=search_query&locale=en&_set_bev_on_new_domain=1612519553_NTk3ZTMyZGU4NzI2> (referer: None)   

Actually, airbnb loads prices for each listing with the domain having .com. I tried various solution like don_redirect etc. but haven't received required response.

来源:https://stackoverflow.com/questions/66064965/stop-scrapy-from-redirecting-to-country-specific-domain

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!