Scraping hover over figure using Python and Selenium

问题

I am trying to scrape the data from http://fuelinsights.gasbuddy.com/Charts using Python and Selenium. The difficult part is that the data only appear when a point on the line graph is hovered over. Currently, my issue is an inability to create a list of all the hover over objects. My code so far is below:

from selenium import webdriver as web
from selenium.webdriver.common.action_chains import ActionChains

driver = web.Chrome('driver path')

driver.get('http://fuelinsights.gasbuddy.com/Charts')

test= driver.find_elements_by_xpath('//*[@class="highcharts-markers"]')

print(test)

which gives me test=[]. Previously, I have used beautifulsoup for all of my scraping projects, but I have redone some of my previous projects to make sure that I understand how Selenium works and haven't had issues.

If anyone can help me solve this issue so I can create a list of the items that I can use ActionChains to hover over and extract the price and date from it would be much appreciated.

Thank you!

****EDIT**** To clarify, I have looked over numerous other posts concerning SVG and g elements and Highcharts, but I am still short on a solution to this problem. I have tried numerous Xpaths (and other find_elements_by options), but have only been able to come to two results: (1) the Xpath is valid, but does not contain any elements, or (2) InvalidSelectorException indicating that I was unable to locate an element with the xpath expression. I believe this comes down to simply incorrectly specifying my Xpath, but I am at a loss for how to find the correct Xpath.

回答1:

You can't use the Xpath which you have mentioned above for locating the elements inside the svg tag.

Xpath which you can use to create a list of hover objects is:

"//[name()='svg']//[name()='g' and @class='highcharts-markers']/*[name()='path']"

I have written a java program for getting the text of all the tool-tip elements. You can use the logic and write a corresponding python code:

1. Get List of tooltip Elements

 List <WebElement> highChartElements= driver.findElements(By.xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-markers']/*[name()='path']"));

2. Iterate through the list and use action class for moving and clicking on all the tooltip Elements

3. Get the text of the tooltip elements.

for(WebElement element:highChartElements){
        Actions action = new Actions(driver);
        action.moveToElement(element).click().perform();
        Thread.sleep(3000);
        List<WebElement> highChartToolTipTextElements= driver.findElements(By.xpath("//*[name()='svg']//*[name()='g' and @class='highcharts-tooltip']/*[name()='text']/*[name()='tspan']"));
        for(WebElement toolTipElement:highChartToolTipTextElements){
            System.out.println("The text for the elements is"+toolTipElement.getText());
        }
    }

来源：https://stackoverflow.com/questions/48180075/scraping-hover-over-figure-using-python-and-selenium

标签

python

selenium

hover