UrlFetchApp.fetch() error, doesn't seem to be using headers

问题

Trying to grab data from a website using Google Apps Script to put it directly into a spreadsheet. The fetch does not seem to be working, where the Python requests equivalent works just fine.

Python code:

page = requests.get("someurl?as_data_structure", headers={'user-agent':'testagent'})

GAS code:

var page = UrlFetchApp.fetch("someurl?as_data_structure", headers={'user-agent':'testagent'});

The only required header is the user-agent, and the error I am getting from the GAS code is what I would usually get from the Python code if I hadn't included the header. I am new to js but as far as I know this is the proper way to do it..?

EDIT: Now got the headers in the right place but the issue persists, exactly the same error as before.

var options = {"headers": {"User-Agent": "testagent"}};
var page = UrlFetchApp.fetch("someurl?as_data_structure", options);

回答1:

Google doesn't always reveal it's restrictions(Annoying?). One such restriction is changing the user agent. It's fixed to

"User-Agent": "Mozilla/5.0 (compatible; Google-Apps-Script)"

You can't change it.

Sample Test:

function testUrlFetchAppHeaders() {
  var options = {
    headers: {
      'User-Agent':
        'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36',
    },
  };
  var fakeRequest = UrlFetchApp.getRequest(
    'https://www.httpbin.org/headers',
    options
  );//providing fake assurance
  var realRequest = UrlFetchApp.fetch(
    'https://www.httpbin.org/headers',
    options
  );//like a wrecking ball
  Logger.log({ fake: fakeRequest, real: realRequest });
}

Sample Response:

{
  "fake": {
    "headers": {
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
    },
    "method": "get",
    "payload": "",
    "followRedirects": true,
    "validateHttpsCertificates": true,
    "useIntranet": false,
    "contentType": null,
    "url": "https://www.httpbin.org/headers"
  },
  "real": {
    "headers": {
      "Accept-Encoding": "gzip,deflate,br",
      "Host": "www.httpbin.org",
      "User-Agent": "Mozilla/5.0 (compatible; Google-Apps-Script)"
    }
  }
}

getRequest(url)

Returns the request that would be made if the operation was invoked.

This method does not actually issue the request.

Neither does it accurately return the request that would be made.

回答2:

the headers belong into the options:

var options = {"headers": {"User-Agent": "testagent"}};
var page = UrlFetchApp.fetch("someurl?as_data_structure", options);

来源：https://stackoverflow.com/questions/56099139/urlfetchapp-fetch-error-doesnt-seem-to-be-using-headers

标签

google-apps-script

http-headers

user-agent

http-status-code-403

urlfetch