Scrapping of the Twitter follower page using selenium and phantomjs

≡放荡痞女 提交于 2019-12-07 15:27:39

问题


I am trying to get the follower list of a given Twitter account. To do so, I manage to connect to my account using Selenium and PhantomJS (v2.1.1). Then, since Twitter is using infinite scrolling, I would like to implement the following algorithm:

1/ Count the number of html elements with class attribute equals ProfileCard-screennameLink 

2/ Repeat:
    -Scroll down
    -Count the number of html elements with class attribute equals ProfileCard-screennameLink
Until the number of elements does not change

My problem is that the scrolling function does not work. Indeed, when I am running

    elemsCount = browser.execute_script("return document.querySelectorAll('.ProfileCard-screennameLink').length")
    print("c", elemsCount)

    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(10)
    elemsCount = browser.execute_script("return document.querySelectorAll('.ProfileCard-screennameLink').length")
    print("c (new)", elemsCount)

There is no change in the number of retrieved HTML elements (neither in the height of the page).. It looks like the scrollTo function has no impact. Do you have any explanation or workaround to solve my problem please?

来源:https://stackoverflow.com/questions/43983874/scrapping-of-the-twitter-follower-page-using-selenium-and-phantomjs

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!