Fetching all links in a website using Selenium

问题

I want to fetch all the links in a website and click each one to those. Basically I want to check if that website have any broken links or not?

One way I was thinking is to add all the links in home map with key as its text and keep on adding in map till all the links in menu item pages are added and then iterate on map and click on each link.

But I know this is not the best solution as we can still miss some links.

Does anybody have any other better solution to this?

回答1:

You can use the below method to verify links are broken or not.

try {
            List<WebElement> anchorTagsList = driver.findElements(By.tagName("a"));
            System.out.println("Total no. of links are " + anchorTagsList.size());
            for (WebElement anchorTagElement : anchorTagsList) {
                if (anchorTagElement != null) {
                    verifyURLStatus(anchorTagElement.getAttribute("href"));
                }
            }

        } catch (Exception e) {
            e.printStackTrace();
            System.out.println(e.getMessage());
        }
    }

    public static List<String> verifyURLStatus(String URL) throws Exception {
        List<String> invalidURL = new ArrayList<String>();
        try {
            URL url = new URL(URL);
            HttpURLConnection httpURLConnect = (HttpURLConnection) url.openConnection();
            httpURLConnect.setConnectTimeout(3000);
            httpURLConnect.connect();
            if (httpURLConnect.getResponseCode() == 200) {
                System.out.println("Valid URL " + URL);
            } else {
                invalidURL.add(URL);
                System.out.println("In Valid URL " + URL);
            }
        } catch (Exception e) {
            e.printStackTrace();
            throw new Exception("Failed to validate broken link");
        }
        return invalidURL;
    }

回答2:

Pradeep's answer is good for validating links, but you seem to be asking how to -get- the links in the first place.

List<WebElement> allLinks = driver.findElements(By.xpath("//a"));

If you were to then use Pradeep's method, you'd need to pass the URL's as strings:

for(WebElement link : allLinks)
    verifyURLStatus(link.getAttribute("href"));

My example ignores the returned list array of invalid URL's, but you get the idea.

来源：https://stackoverflow.com/questions/48555407/fetching-all-links-in-a-website-using-selenium

标签

selenium

selenium-chromedriver