How to optimize the scroll-down code in java using selenium

问题

I am working a project in MAVEN using Java. I have to get a URL, scroll them down ,and get all the links of other items in this given web page.

Till now, I get the page dynamically using Selenium , and scrolling them down, and fetch the links also. But it takes too much time. Please help me in optimize that.

Example:-, I am working on a page , whose link is here.

My Questions :-

Scrolling web page using selenium is very slow. How can I optimize this? (Suggest any other method
to do the same or help me to optimize this one)

Thanks in advance. Looking for your kind response.

Code to dynamically get and scroll the page:-

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
import com.google.common.collect.*;
import java.io.File;
import java.util.ArrayList;
import java.util.Date;
import org.apache.commons.io.FileUtils;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxProfile;

/**
 *
 * @author jhamb
 */
public class Scroll_down {

    private static FirefoxProfile createFirefoxProfile() {
        File profileDir = new File("/tmp/firefox-profile-dir");
        if (profileDir.exists()) {
            return new FirefoxProfile(profileDir);
        }
        FirefoxProfile firefoxProfile = new FirefoxProfile();
        File dir = firefoxProfile.layoutOnDisk();
        try {
            profileDir.mkdirs();
            FileUtils.copyDirectory(dir, profileDir);
        } catch (IOException e) {
            e.printStackTrace();
        }
        return firefoxProfile;
    }



    public static void main(String[] args)  throws InterruptedException{
        String url1 = "http://www.jabong.com/men/shoes/men-sports-shoes/?source=home-leftnav";
        System.out.println("Fetching %s..." + url1);
        WebDriver driver = new FirefoxDriver(createFirefoxProfile());


        driver.get(url1);  

        JavascriptExecutor jse = (JavascriptExecutor)driver;
        jse.executeScript("window.scrollBy(0,250)", "");
        for (int second = 0;; second++) {
            if (second >= 60) {
                break;
            }
            jse.executeScript("window.scrollBy(0,200)", "");
            Thread.sleep(1000);
        }
            String hml = driver.getPageSource();
        driver.close();


        Document document = Jsoup.parse(hml);

            Elements links = document.select("div");

        for (Element link : links) {
            System.out.println(link.attr("data-url"));
        }
    }
}

回答1:

Well Selenium scrolling is based on Javascript. I dont know your goal with selenium though, you have no assertion to compare anything in your code ? When you are so sure that your data fetching so fast then don't use any sleep methode. Sleep methods makes selenium slower, but yeah it is waiting until the element is properly loaded ..... It's up to you, what to test though

回答2:

How about page down?

ele.sendKeys(Keys.PAGE_DOWN);   //WebElement ele = <Any existing element>

Repeat this till you find that particular item.

来源：https://stackoverflow.com/questions/15919602/how-to-optimize-the-scroll-down-code-in-java-using-selenium

标签

java

javascript

selenium

web-crawler

jsoup