How can I input data into a webpage to scrape the resulting output using Python?

Question

I am familiar with BeautifulSoup and urllib2 to scrape data from a webpage. However, what if a parameter needs to be entered into the page before the result that I want to

Answer 1

Take a look at tools like mechanize or scrape:

• http://pypi.python.org/pypi/mechanize
• http://stockrt.github.com/p/emulating-a-browser-in-python-with-mechanize/

• http://www.ibm.com/developerworks/linux/library/l-python-mechanize-beautiful-soup/

• http://zesty.ca/scrape/

Packt Publishing has an article on that matter, too:

• http://www.packtpub.com/article/web-scraping-with-python