How can I make Selenium click on the “Next” button until it is no longer possible?

Question

I would like to write a code that would make Python scrape some data on a page, then click on the "next" button at the bottom of the page, scrape some data on the second page, click on the "next" button, etc. until the last page, where clicking on "Next" is no longer possible (because there is no "next").

I would like to make the code as general as possible and not specify beforehand the number of clicks to be done. Following this question (How can I make Selenium click through a variable number of "next" buttons?), I have the code below. Python does not report any error, but the program stops after the first iteration (after the first click on the "next").

What am I missing here? Many thanks!

driver = webdriver.Firefox()
driver.get("http://www.mywebsite_example.com")
try:
    wait = WebDriverWait(driver, 100)
    wait.until(EC.element_to_be_clickable((By.CLASS_NAME,'reviews_pagination_link_nav')))    
    driver.find_element_by_class_name("reviews_pagination_link_nav").click()

    wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'reviews_pagination_link_nav')))
    while EC.element_to_be_clickable((By.CLASS_NAME,'reviews_pagination_link_nav')):
      driver.find_element_by_class_name("reviews_pagination_link_nav").click()
      if not driver.find_element_by_class_name("reviews_pagination_link_nav"):
        break
      wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'reviews_pagination_link_nav')))

finally:
    driver.quit()

Answer 1

I would make an endless while True loop and break it once there is TimeoutException thrown - this would mean there are no pages to go left:

wait = WebDriverWait(driver, 10)
while True:
    # grab the data

    # click next link
    try:
        element = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'reviews_pagination_link_nav')))
        element.click()
    except TimeoutException:
        break

For this to work, you need to make sure that once you hit the last page, the element with class="reviews_pagination_link_nav" is not on the page or is not clickable.