Scroll to the end of the infinite loading page using selenium Python

问题

I'm scraping the follower names from the twitter using Selenium and that page is infinite, whenever I scroll down I can see new followers. Somehow I want to go to the bottom of the page so that I can scrap all the followers.

while number != 5:
   driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")
   number = number + 1
   time.sleep(5)

usernames = driver.find_elements_by_class_name(
       "css-4rbku5.css-18t94o4.css-1dbjc4n.r-1loqt21.r-1wbh5a2.r-dnmrzs.r-1ny4l3l")
for username in usernames:
   print(username.get_attribute("href"))

Right now code is scrolling 5 times. I have put a static value but I don't know how much scrolls are needed to reach the bottom of the page.

回答1:

Use below code for infinite loading. It will keep scrolling until new elements are getting loaded i.e. page size is changing.

# Get scroll height after first time page load
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    # Wait to load page
    time.sleep(2)
    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

来源：https://stackoverflow.com/questions/63647849/scroll-to-the-end-of-the-infinite-loading-page-using-selenium-python