UrlFetchApp not connecting to public URL

问题

I'm trying to pull data from this website using the download CSV link found under Actions > Download > CSV

https://app.udot.utah.gov/apex/eom/f?p=288:300

I get an error 302 on the UrlFetchApp line consistently and I'm not understanding why. I ran the logger with the httpexception and got the error that it could not connect to the server. The website is public and accessible regardless of network so I'm having a hard time understanding what's going on. The URL seems to download the CSV when input manually but I cannot get the UrlFetchApp to grab it.

var EquipUrl = "https://app.udot.utah.gov/apex/eom/f?p=288:300:::CSV";
var EquipContent = UrlFetchApp.fetch(EquipUrl).getContentText();
var EquipData = Utilities.parseCsv(EquipContent);

Any help would be appreciated!

I just tried the following per suggestion in comments and had the same Error 302 result. I also tried setting redirects to false and this proceeds and ends with an empty EquipData array

var options = {
'followRedirects' :true
];
var EquipUrl = "https://app.udot.utah.gov/apex/eom/f?p=288:300:::CSV::::";
var EquipContent = UrlFetchApp.fetch(EquipUrl, options).getContentText();
var EquipData = Utilities.parseCsv(EquipContent);

Thanks for the help so far TheMaster! I said in my comment that the code was spitting out an error on the var cookies line but that error rectified itself so I think it may have been a bad copy of your code on my part. The problem is now that the final Logger line spits out a "Could not parse text" error.

function fetchUrlWithCookie() {
  var url = 'https://app.udot.utah.gov/apex/eom/f?p=288:300:::CSV';
  var response = UrlFetchApp.fetch(url, {
    muteHttpExceptions: true,
    followRedirects: false,
  });
  var cookie = response.getAllHeaders()['Set-Cookie'];
  response = UrlFetchApp.fetch(url, {
    muteHttpExceptions: true,
    headers: {
      Cookie: cookie, //send the cookie we got as header
    },
  });
  Logger.log(Utilities.parseCsv(response.getContentText()));//parseCSV
}

I tried a few different options like trying to log before to see if I could find anything. Is there any information I could pull from Stackdriver logging that could help identify issues?

Thank you so much TheMaster! It worked! I would love if you could comment and address your problem solving process so I can help learn what to be on the lookout for next time this happens but I know that's asking a lot.

  var url = 'https://app.udot.utah.gov/apex/eom/f?p=288:300::CSV::::';
  var response = UrlFetchApp.fetch(url, {
    muteHttpExceptions: true,
    followRedirects: true,
  });
  var cookie = response.getAllHeaders()['Set-Cookie']; 
  response = UrlFetchApp.fetch(url, {
    muteHttpExceptions: true,
    headers: {
      Cookie: cookie, //send the cookie we got as headerw
    },
  });
  Logger.log(Utilities.parseCsv(response.getContentText()));//parseCSV

That's a copy of the code that fully functions. Thank you again for your help over this several day period to address this issue.

回答1:

Key Points:

Some websites use cookies for maintaining state, session and completing the request. Since urlFetch runs on server side and not in a browser, cookies are not maintained across requests. However, We can manually get the cookie in the first request and send it in subsequent requests.

Code Snippet:

function fetchUrlWithCookie() {
  var url = 'xxxx'; //Your csv url
  var response = UrlFetchApp.fetch(url, {
    muteHttpExceptions: true,
    followRedirects: false,
  });
  var cookie = response.getAllHeaders()['Set-Cookie']; //Get cookie from header 
  response = UrlFetchApp.fetch(url, {
    muteHttpExceptions: true,
    headers: {
      Cookie: cookie, //send the cookie we got as header
    },
  });
  Logger.log(Utilities.parseCsv(response.getContentText()));//parseCSV
}

References:

• RFC6265
• UrlFetchApp
• Response headers

来源：https://stackoverflow.com/questions/53660031/urlfetchapp-not-connecting-to-public-url