How to load all entries in an infinite scroll at once to parse the HTML in python

Question

I am trying to extract information from this page. The page loads 10 items at a time, and I need to scroll to load all entries (for a total of 100). I am able to parse the H

Answer 1

Alternatively, you could try the answer given by Andrew Che in this question. This is a 3-year-old question, and the page is not there anymore, but if you try something similar with a scrolling results page (e.g. I searched for 'top'), you could find out the endpoint you can call to get the different pages in the results list, which in this case was 'https://medium.com/search/posts?q=top&page=1'.