Selenium: retrieve data that loads while scrolling down

Question

I\'m trying to retrieve elements in a page that has an ajax-load scroll-down functionality alla Twitter. For some reason this isn\'t working properly. I added some print sta

Answer 1

The condition in the while loop was the issue for my use case. It was an infinite loop. I fixed the problem by using a counter :

def get_items(items):

    item_nb = [0, 1] # initializing a counter of number of items found in page

    while(item_nb[-1] > item_nb[-2]):   # exiting the loop when no more new items can be found in the page

        items = wd.find_elements_by_class_name('stream-item')
        time.sleep(5)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        item_nb.append(len(items))

    return items

```