How can I access this type of site using requests? [duplicate]

问题

This is the first time I've encountered a site where it wouldn't 'allow me access' to the webpage. I'm not sure why and I can't figure out how to scrape from this website.

My attempt:

import requests
from bs4 import BeautifulSoup

def html(url):
    return BeautifulSoup(requests.get(url).content, "lxml")

url = "https://www.g2a.com/"

soup = html(url)

print(soup.prettify())

Output:

<html>
 <head>
  <title>       
   Access Denied
  </title>
 </head>
 <body>
  <h1>
   Access Denied
  </h1>
  You don't have permission to access "http://www.g2a.com/" on this server.
  <p>
   Reference #18.4d24db17.1592006766.55d2bc1
  </p>
 </body>
</html>

I've looked into it for awhile now and I found that there is supposed to be some type of token [access, refresh, etc...].

Also, action="/search" but I wasn't sure what to do with just that.

回答1:

This page needs to specify some HTTP headers to obtain the information (Accept-Language):

import requests
from bs4 import BeautifulSoup

headers = {'Accept-Language': 'en-US,en;q=0.5'}

def html(url):
    return BeautifulSoup(requests.get(url, headers=headers).content, "lxml")

url = "https://www.g2a.com/"

soup = html(url)

print(soup.prettify())

Prints:

<!DOCTYPE html>
<html lang="en-us">
 <head>
  <link href="polyfill.g2a.com" rel="dns-prefetch"/>
  <link href="images.g2a.com" rel="dns-prefetch"/>
  <link href="id.g2a.com" rel="dns-prefetch"/>
  <link href="plus.g2a.com" rel="dns-prefetch"/>

... and so on.

来源：https://stackoverflow.com/questions/62354101/how-can-i-access-this-type-of-site-using-requests

标签

python

web-scraping

python-requests