Python Selenium - Clicking Next Page

问题

I am currently writing a python selenium script to take information off a website. I have successfully got the data off page 1 of 100+ in the format I want. I unfortunately can’t get the program to run and collect all the information off the proceeding pages. When I look at the web site target=https://www.freshfields.com/en-gb/contacts/find-a-lawyer/ script it shows me that the “Next” button is compiled like the below;

/body/div[@id='main-content']/div[@class='t6a-grid']/div[@class='mmargin-bottom-30']/div[@id='grid']/div[@class='row-margin-bottom-10']/div[@class='col-md-12 padding-left-0 padding-right-20']/ul[@class='pagination']/li[11]/a Part of the script I have written is below. The "# this is navigate to next page element" in the script is the area that isn’t currently working.

def get_links(driver, target): # this is to collect links that associate with all the profiles present in Freshfields website

driver.get(target)
# get links associated to profiles on result page
list_links = [] 
while True:
    list_ppl_link =  driver.find_elements_by_xpath('//div[@class=" mix item col-xs-6 col-sm-4"]')      
    for item in list_ppl_link:
        emp_name_obj = item.find_element_by_tag_name('a')
        emp_name = emp_name_obj.text
        emp_link = emp_name_obj.get_attribute('href')
        list_links.append({'emp_name':emp_name, 'emp_link':emp_link})
    try:
        # this is navigate to next page
        driver.find_element_by_xpath('//ul[@class="pagination"]/li').click()
        time.sleep(1)
    except NoSuchElementException:
        break
return list_links

Please could somebody help me to understand how I can loop through the pages and collect the 1,960 records?

Many thanks

Chris

回答1:

try using something like below:

list_ppl_link =  driver.find_elements_by_xpath('//div[@class=" mix item col-xs-6 col-sm-4"]') 
    i=1     
    for item in list_ppl_link:
        i=i+1
        emp_name_obj = item.find_element_by_tag_name('a')
        emp_name = emp_name_obj.text
        emp_link = emp_name_obj.get_attribute('href')
        list_links.append({'emp_name':emp_name, 'emp_link':emp_link})
    try:
        # this is navigate to next page
        driver.find_element_by_xpath('//ul[@class="pagination"]//li/a[contains(text(),"' + str(i) +'")').click()
        time.sleep(1)
    except NoSuchElementException:
        break

来源：https://stackoverflow.com/questions/43886907/python-selenium-clicking-next-page