Java parsing html elements generated by JS

Question

I\'m very new to html parsing with Java, I used JSoup previously to parse simple html without it dynamically changing, however I now need to parse a web page that has dynami

Answer 1

The problem you are facing is Jsoup retrieves the static source code, as it would be delivered to a browser. What you want is the DOM after the javaScript has been invoked. For this, you can use HTML Unit to get the rendered page and then pass its content to Jsoup for parsing.

// capture rendered page
WebClient webClient = new WebClient();
HtmlPage myPage = webClient.getPage("https://pokevision.com");

// convert to jsoup dom
Document doc = Jsoup.parse(myPage.asXml());

// extract data using jsoup selectors
Elements images = doc.select("img[src~=(?i)\\.(png|jpe?g|gif)]");
for (Element image : images) {
    System.out.println("src : " + image.attr("src"));
}

// clean up resources
webClient.close();