POST request works in Postman, but not in Python Requests (200 response with robot detection)

问题

I have a POST request that works perfectly with both Postman an cURL (it returns a JSON blob of data). However, when I perform the exact same request with Python's Requests library, I get a 200 success response, but instead of my JSON blob, I get this:

<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>

I've used HTTP request bins to verify that the request from Postman/cURL is exactly the same as the one from Python Requests.

Here is my Postman request in cURL:

curl -X POST \
  https:/someurl/bla/bla \
  -H 'Content-Type: application/json' \
  -H 'Postman-Token: 2488e914-531e-4ac7-ae8d-8490b2242396' \
  -H 'Referer: https://www.host.com/bla/bla/' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:65.0) Gecko/20100101 Firefox/65.0' \
  -H 'cache-control: no-cache' \
  -d '{"json1":"blabla","etc":"etc"}'

...and here is my Python code:

payload = {
      "json1": "blabla",
      "etc": "etc",
    }

    headers = {
        'Host': 'www.host.com',
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36',
        'Accept': 'application/json, text/javascript, */*; q=0.01',
        'Accept-Language': 'en-US,en;q=0.5',
        'Accept-Encoding': 'gzip, deflate, br',
        'Referer': 'https://www.host.com/bla/bla/', 
        'Content-Type':'application/json',
        'X-Requested-With': 'XMLHttpRequest',
        'Connection': 'keep-alive',
        'Origin': 'https://www.host.com',
    }

    s = requests.Session()
    response_raw = s.post(url, json=payload, headers=headers)
    print(response_raw)
    print(response_raw.text)

I have verified that the payload and headers are correct and valid. Any help would be much appreciated; thanks!

回答1:

You are getting a 200 success response but not JSON data in the response.
This means that is just a response object. It contains only response code
to extract blob information from the response, convert response object to json
simply json_resp = response_raw.json()
This json_resp contains your actual response details.

回答2:

I had a similar issue that I was able to resolve by sending a cookie in the request. Try this:

...
my_cookie = {"Cookie": "cookie text..."}

s = requests.Session()
response_raw = s.post(url, json=payload, headers=headers, cookies=my_cookie)
print(response_raw)
print(response_raw.text)
print(response_raw.content)

You can grab the cookie from the Network tab in the browser's Dev Tools console in the Request Headers section. It sounds like you may also be able to get the cookie using Python's CookieJar lib.

来源：https://stackoverflow.com/questions/54878769/post-request-works-in-postman-but-not-in-python-requests-200-response-with-rob

标签

curl

web-scraping

python-requests

postman

incapsula