How can I input data into a webpage to scrape the resulting output using Python?

前端 未结 5 1499
予麋鹿
予麋鹿 2020-12-15 12:43

I am familiar with BeautifulSoup and urllib2 to scrape data from a webpage. However, what if a parameter needs to be entered into the page before the result that I want to

5条回答
  •  北海茫月
    2020-12-15 13:30

    I think you can also use PySide/PyQt, because they have a browser core of qtwebkit, you can control the browser to open pages, simulate human actions(fill, click...), then scrape data from pages. FMiner is work on this way, it's a web scraping software I developed with PySide.

    Or you can try phantomjs, it's an easy library to control browser, but not it's javascript not python lanuage.

提交回复
热议问题