Clicking multiple items on one page using selenium

问题

My main purpose is to go to this specific website, to click each of the products, have enough time to scrape the data from the clicked product, then go back to click another product from the page until all the products are clicked through and scraped (The scraping code I have not included).

My code opens up chrome to redirect to my desired website, generates a list of links to click by class_name. This is the part I am stuck on, I would believe I need a for-loop to iterate through the list of links to click and go back to the original. But, I can't figure out why this won't work.

Here is my code:

import csv
import time
from selenium import webdriver
import selenium.webdriver.chrome.service as service
import requests
from bs4 import BeautifulSoup


url = "https://www.vatainc.com/infusion/adult-infusion.html?limit=all"
service = service.Service('path to chromedriver')
service.start()
capabilities = {'chrome.binary': 'path to chrome'}
driver = webdriver.Remote(service.service_url, capabilities)
driver.get(url)
time.sleep(2)
links = driver.find_elements_by_class_name('product-name')


for link in links:
    link.click()
    driver.back()
    link.click()

回答1:

I have another solution to your problem.

When I tested your code it showed a strange behaviour. Fixed all problems that I had using xpath.

url = "https://www.vatainc.com/infusion/adult-infusion.html?limit=all"
driver.get(url)
links = [x.get_attribute('href') for x in driver.find_elements_by_xpath("//*[contains(@class, 'product-name')]/a")]
htmls = []
for link in links:
    driver.get(link)
    htmls.append(driver.page_source)

Instead of going back and forward I saved all links (named as links) and iterate over this list.

来源：https://stackoverflow.com/questions/53398126/clicking-multiple-items-on-one-page-using-selenium

标签

python

selenium

beautifulsoup