python-requests

How to deal with 401 (unauthorised) in python requests

由 爷，独闯天下 提交于 2021-01-18 06:23:05
问题 What I want to do is GET from a site and if that request returns a 401, then redo my authentication wiggle (which may be out of date) and try again. But I don't want to try a third time, since that would be my authentication wiggle having the wrong credentials. Does anyone have a nice way of doing this that doesn't involve properly ugly code, ideally in python requests library, but I don't mind changing. 回答1: It doesn't get any less ugly than this, I think: import requests from requests.auth

How to scrape Instagram with BeautifulSoup

由 回眸只為那壹抹淺笑 提交于 2021-01-16 08:11:55
问题 I want to scrape pictures from a public Instagram account. I'm pretty familiar with bs4 so I started with that. Using the element inspector on Chrome, I noted the pictures are in an unordered list and li has class 'photo', so I figure, what the hell -- can't be that hard to scrape with findAll, right? Wrong: it doesn't return anything (code below) and I soon notice that the code shown in element inspector and the code that I drew from requests were not the same AKA no unordered list in the

How to scrape Instagram with BeautifulSoup

由 梦想与她 提交于 2021-01-16 08:00:12
问题 I want to scrape pictures from a public Instagram account. I'm pretty familiar with bs4 so I started with that. Using the element inspector on Chrome, I noted the pictures are in an unordered list and li has class 'photo', so I figure, what the hell -- can't be that hard to scrape with findAll, right? Wrong: it doesn't return anything (code below) and I soon notice that the code shown in element inspector and the code that I drew from requests were not the same AKA no unordered list in the

How to scrape Instagram with BeautifulSoup

由 无人久伴 提交于 2021-01-16 07:59:16
问题 I want to scrape pictures from a public Instagram account. I'm pretty familiar with bs4 so I started with that. Using the element inspector on Chrome, I noted the pictures are in an unordered list and li has class 'photo', so I figure, what the hell -- can't be that hard to scrape with findAll, right? Wrong: it doesn't return anything (code below) and I soon notice that the code shown in element inspector and the code that I drew from requests were not the same AKA no unordered list in the

How to scrape Instagram with BeautifulSoup

由 泄露秘密 提交于 2021-01-16 07:52:55
问题 I want to scrape pictures from a public Instagram account. I'm pretty familiar with bs4 so I started with that. Using the element inspector on Chrome, I noted the pictures are in an unordered list and li has class 'photo', so I figure, what the hell -- can't be that hard to scrape with findAll, right? Wrong: it doesn't return anything (code below) and I soon notice that the code shown in element inspector and the code that I drew from requests were not the same AKA no unordered list in the

How can I check if either xpath exists and then return the value if text is present?

由 自闭症网瘾萝莉.ら 提交于 2021-01-07 01:44:38
问题 I'm having trouble with the second r.html.xpath request. When there is a special deal on an item, the second Xpath changes from //*[@id="priceblock_ourprice"] to //*[@id="priceblock_dealprice"] This causes the script to fail since there the right xpath cannot be returned. How can I include this second xpath that only shows up occasionally? I would like to see if either xpath exists, if so return that, or return N/A. The first url that is searched has the ourprice xpath and the second url has

How can I check if either xpath exists and then return the value if text is present?

由 对着背影说爱祢 提交于 2021-01-07 01:43:10
问题 I'm having trouble with the second r.html.xpath request. When there is a special deal on an item, the second Xpath changes from //*[@id="priceblock_ourprice"] to //*[@id="priceblock_dealprice"] This causes the script to fail since there the right xpath cannot be returned. How can I include this second xpath that only shows up occasionally? I would like to see if either xpath exists, if so return that, or return N/A. The first url that is searched has the ourprice xpath and the second url has

python web scraping request error(mod security)

由 别说谁变了你拦得住时间么 提交于 2021-01-06 02:48:10
问题 I am new and I try to grap source code of an Web page for tutorial.I got beautifulsoup install,request install. At first I want to grap the source.I am doing this scraping job from "https://pythonhow.com/example.html".I am not doing anything illegal and I think this site also established for this purposes.Here's my code: import requests from bs4 import BeautifulSoup r=requests.get("http://pythonhow.com/example.html") c=r.content c And i got the mod security error: b'<head><title>Not

python web scraping request error(mod security)

由 一个人想着一个人 提交于 2021-01-06 02:43:25
问题 I am new and I try to grap source code of an Web page for tutorial.I got beautifulsoup install,request install. At first I want to grap the source.I am doing this scraping job from "https://pythonhow.com/example.html".I am not doing anything illegal and I think this site also established for this purposes.Here's my code: import requests from bs4 import BeautifulSoup r=requests.get("http://pythonhow.com/example.html") c=r.content c And i got the mod security error: b'<head><title>Not

Creating an API to execute a python script

由 馋奶兔 提交于 2021-01-05 11:11:24
问题 I have a python script app.py in my local server ( path =Users/soubhik.b/Desktop) that generates a report and mails it to certain receivers. Instead of scheduling this script on my localhost, i want to create an API which can be accessed by the receivers such that they would get the mail if they hit the API with say a certain id. With the below code i can create an API to display a certain text. But, what do i modify to run the script through this? Also if i want to place the script in a
订阅 python-requests