How can I use Perl to scrape a website that reveals its content with JavaScript?

Question

I need to write a Perl script to scrape a website. The website can only be scraped with JavaScript, and the user is on Windows.

I got some way with Win32::IE::Mech

Answer 1

WWW::Selenium.

• It allows you to specify which browser to use (IE and Firefox are supported from the get-go)
• It supports access to elements via xpath elements, table IDs, text (regex-matching!) and URLs
• It provides a Swiss army knife of user-interaction options, giving you flexibility over how you wish to simulate end-user browsing

You'll need to download the Selenium Remote Control and have it running in the background for the module to work.

It may not be a good option if your page load times are unpredictable.