Clicking “More” button via selenium

Question

I am trying to scrape the following website : https://angel.co/companies

There is a \"More\" button at the bottom, which on click loads more records.

I need

Answer 1

You are doing right, just wait a little. The ajax triggers after selenium callback. You can write something like this or use "assert":

button = None
while not button:
    button = driver.find_element_by_class_name("more")
    if button:
        break

You also can try to use ajaxes instead of selenium. Try this url changing page parameter:

https://angel.co/companies/startups?ids[]=81494&ids[]=3322647&ids[]=98145&ids[]=32119&ids[]=21604&ids[]=19935&ids[]=480579&ids[]=3062473&ids[]=431924&ids[]=395542&ids[]=154&ids[]=948481&ids[]=197974&ids[]=891681&ids[]=972236&ids[]=686564&ids[]=115616&ids[]=515341&ids[]=1856&ids[]=477880&total=4381226&page=3&sort=signal&new=false&hexdigest=be1927797c1b88f79ae42efd4180ea78d3e9e711

Look, the website returns json file with dictionary of one key - "html", this is the htmlcode servers return.