Getting all href from a code

问题

I'm making a web-crawler. For finding the links in a page I was using xpath in selenium

driver = webdriver.Firefox()
driver.get(side)
Listlinker = driver.find_elements_by_xpath("//a")

This worked fine. Testing the crawler however, I found that not all links come under the a tag. href is sometimes used in area or div tags as well.

Right now I'm stuck with

driver = webdriver.Firefox()
driver.get(side)
Listlinkera = driver.find_elements_by_xpath("//a")
Listlinkerdiv = driver.find_elements_by_xpath("//div")
Listlinkerarea = driver.find_elements_by_xpath("//area")

which really puts the crawl in web-crawler.

I've tried xpath "//@href", but that doesn't work. I've also tried several ways to get all href url's in an efficient manner, both using beautiful soup and lxml, but so far, to no avail. I'm sorry I do not have any code to show for my efforts with beautiful soup and lxml, but as these proved useless, I deleted them, which isn't the smartest practice, I know. I have now started to save these unsuccessful attempts, for my own sake, if I ever want to try again, and want to know what went wrong the first time

Any help I could get on this would be greatly appreciated.

回答1:

Try this:

ListlinkerHref = driver.find_elements_by_xpath("//*[@href]")

来源：https://stackoverflow.com/questions/8572540/getting-all-href-from-a-code