Headless chrome with selenium, can only find ways to scroll non-headless

Question

There is a lot to find about this subject, but can't figure this out. I need to scroll to the end of the page of a (not so long) infinity scroll. I have 2 options that work with chrome non-headless but doesn't seem to work headless.

The first one which i liked the most, works beautiful and found here on SA:

driver = webdriver.Chrome('c:/cd.exe', chrome_options=chrome_options)
driver.get('http://www.website.com')

while True:
    count = len(driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]'))
    print(count)
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    try:
        WebDriverWait(driver, 50).until(EC.visibility_of_element_located((By.XPATH,
                                                                          "//div[@itemprop='itemListElement'][%s]" % str(count + 1))))
    except TimeoutException:
        break

Second more hack job after realizing i can't get away with above in headless mode:

driver = webdriver.Chrome('c:/cd.exe', chrome_options=chrome_options)
driver.get('https://www.website.com')

while True:

    count = len(driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]'))
    actions = ActionChains(driver)
    actions.send_keys(Keys.PAGE_DOWN)
    actions.perform()
    actions.send_keys(Keys.PAGE_DOWN)
    actions.perform()


    # focus_element_scroll = driver.find_elements_by_xpath('//section[@class="occasion-content"]')
    # driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]')[-1].send_keys(Keys.PAGE_DOWN)
    # driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]')[-1].send_keys(Keys.PAGE_DOWN)
    # self.driver.find_element_by_css_selector("ul.list-with-results").send_keys(Keys.ARROW_DOWN)
    print(count)
    # driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    try:  
        WebDriverWait(driver, 50).until(EC.visibility_of_element_located((By.XPATH,
                                                                          "//div[@itemprop='itemListElement'][%s]" % str(count + 1))))
    except TimeoutException:
        break

So both work in chrome but don't in headless mode, i need to push them to a ubuntu vps where they need to be headless, i know of the xvfb option but i am glad i could remove that and use native chrome since the droplets don't have much memmory.

Edit: Just tried this approach with focus on an element in the footer, also works in non-headless but doesn't in headless:

ActionChains(driver).move_to_element(focus[0]).perform()

Someone got a different approach?

Edit Just want to know if it is possible to scroll with chrome in headless mode!

Answer 1

Found the answer after 2 days trying different combinations of versions of selenium, chrome and chromedriver i alsmost gave up and wanted to go with the xvfb.

Already tried to maximze the window in the chrome arguments, that didn't help. But this time i tried setting a manual window size. That helped.

    chrome_options.add_argument("window-size=1920,1080")

Posting here so that the next one wont take as long as me.

Answer 2

I just ran into this problem on windows. Using chrome 74 and i fixed the issue by having the below chromeOptions. My headless mode works again :) Thanks at DebanjanB

chromeOptions.addArguments("--headless")
chromeOptions.addArguments("--no-sandbox")
chromeOptions.addArguments("--disable-dev-shm-usage")
chromeOptions.addArguments("--window-size=1920x1080")
chromeOptions.addArguments("start-maximised")

Answer 3

To scroll to the end of the page of a (not so long) infinity scroll through Default Chrome Browser and Headless Chrome Browser you can use the following code block :

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

options = Options()
options.add_argument("--headless")
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('http://www.website.com')

while (driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")):
    try:
        WebDriverWait(driver, 50).until(EC.visibility_of_element_located((By.XPATH, "//div[@itemprop='itemListElement']" )))
        # do your other actions within the Viewport
    except TimeoutException:
        break
print("Reached to the bottom of the page")