Getting the final destination of a javascript redirect on a website

Question

I parse a website with python. They use a lot of redirects and they do them by calling javascript functions.

So when I just use urllib to parse the site, it doesn\'t

Answer 1

I looked into Selenium. And if you are not running a pure script (meaning you don't have a display and can't start a "normal" browser) the solution is actually quite simple:

from selenium import webdriver

driver = webdriver.Firefox()
link = "http://yourlink.com"
driver.get(link)

#this waits for the new page to load
while(link == driver.current_url):
  time.sleep(1)

redirected_url = driver.current_url

For my usecase this is more than enough. Selenium can also interact with forms and send keystrokes to the website.