Trouble getting desired response issuing a post requests

问题

I've created a script in python to get a 200 status code issuing a post http requests but when I run my script I get 403 instead. It seems that I followed the way how the requests is being sent in chrome dev tools.

To do it manually - go to that page, select 6 as size and then hit the add to cart button.

How can I do the same using the script below?

Webpage address

I've tried with:

import requests
from bs4 import BeautifulSoup

main_url = 'https://www.footlocker.co.uk/en/homepage'
post_url = 'https://www.footlocker.co.uk/en/addtocart?'

params = {
    'SynchronizerToken': '',
    'Ajax': True,
    'Relay42_Category': 'Product Pages',
    'acctab-tabgroup-314207586604090': None,
    'Quantity_314207586604070': '1',
    'SKU': '314207586604070'
}

with requests.Session() as s:
    r = s.get(main_url)
    soup = BeautifulSoup(r.text,"lxml")

    #parsing token to reuse within data
    token = soup.select_one("[name='SynchronizerToken']")['value']

    params['SynchronizerToken'] = token

    res = s.post(post_url,params=params,data=params,headers={
        'user-agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36',
        'x-requested-with': 'XMLHttpRequest',
        'referer': 'https://www.footlocker.co.uk/en/p/nike-signal-dmsx-men-shoes-73190?v=314207586604',
        'accept': 'application/json, text/javascript, */*; q=0.01'
        })
    print(res.status_code)

Current status:

Expected status:

回答1:

I tested your code and got a 200 response, so I ran your code again in a for loop like this:

for i in range(100):
    with requests.Session() as s:
        r = s.get(main_url)

...

And after the 8th call, I started receiving 403 responses (IE permission denied).

You said you were using a vpn, but it looks like they either ban certain vpn providers, or you are hitting the request limit with a given IP without realizing it.

I assume the former, since I tested your code using tor and also got a 403.

回答2:

I've tested your code, and it looks perfect. Actually, It's not your code's fault. It's from website. After testing your code, I go through the website and tested it manually. And I found that the side will be disabled Add to cart button if you continuously try to hit the Add to cart button around 5-8 times. So, you need to think in that way. Your code is perfect.

回答3:

I tested your code and it works. Received 200.

403 fobidden means

Client : Give me your data
Server: Nope

If you have got 200 status code before, more likely you have been blocked.

Another possibility is that the server side service logic has been modified to protect themselves, in this case, no one can access, but the server itself can internally. But this won't be the case as I got 200.

Each server has own policy to protect itself. Some blocks temporarily, some blocks permanently.

Here what you can try,

1. Change User-Agent Sometime server blocks certain browser which requests too much. Use the latest version of different browsers.

2. Change your IP (Of course changed IP can be blocked)

3. Increase your request interval

来源：https://stackoverflow.com/questions/57827402/trouble-getting-desired-response-issuing-a-post-requests

标签

python

python-3.x

web-scraping