How to load all entries in an infinite scroll at once to parse the HTML in python

Question

I am trying to extract information from this page. The page loads 10 items at a time, and I need to scroll to load all entries (for a total of 100). I am able to parse the H

Answer 1

You can try with this :

from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.support.ui import WebDriverWait

pause = 10
driver = webdriver.PhantomJS(executable_path='phantomjs.exe')
driver.get("your_url")
#This code will scroll down to the end
while True:
     try:
        # Action scroll down
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
     break
 except: 
     pass