Clicking “More” button via selenium

Question

I am trying to scrape the following website : https://angel.co/companies

There is a \"More\" button at the bottom, which on click loads more records.

I need

Answer 1

The more you will click the MORE button more data will be loaded. You need to induce WebDriverWait for the button with text with MORE to be clickable and you can use the following solution:

• Code Block:

  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.common.exceptions import TimeoutException
  
  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_argument('disable-infobars')
  driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  driver.get("https://angel.co/companies")
  WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='more' and contains(.,'More')]")))
  while True:
      try:
          WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='more' and contains(.,'More')]"))).click()
          print("MORE button clicked")
      except TimeoutException:
          break
  driver.quit()
  

• Console Output:

  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked
  MORE button clicked