crawl dynamic web page using htmlunit

后端 未结 2 545
你的背包
你的背包 2020-12-16 01:00

I am crawling data using HtmlUnit from a dynamic webpage, which uses infinite scrolling to fetch data dynamically, just like facebook\'s newsfeed. I used the following sente

相关标签:
2条回答
  • 2020-12-16 01:19

    I had similiar problem where the content were post-loaded during page scrolling. I solved it using:

    webClient.getCurrentWindow().setInnerHeight(Integer.MAX_VALUE);

    0 讨论(0)
  • 2020-12-16 01:29

    I was searching the same thing. I was only able to find that it is not scroll event (90% sure). There is link on JS wich is responsilbe for loading the page and could maybe help you.

    0 讨论(0)
提交回复
热议问题