How can I use Perl to scrape a website that reveals its content with JavaScript?

Question

I need to write a Perl script to scrape a website. The website can only be scraped with JavaScript, and the user is on Windows.

I got some way with Win32::IE::Mech

Answer 1

Have a look at Win32::Watir. It's a newer module and explicitly supports IE 6, 7 and 8.

Answer 2

WWW::Scripter and its ::Plugin::Javascript can probably help you.

Answer 3

I don't see any mention of WWW::Mechanize, so I'll bring it up just for completeness. Selenium is also becoming very popular and can be used in a lot of testing scenarios.

Answer 4

WWW::Selenium.

• It allows you to specify which browser to use (IE and Firefox are supported from the get-go)
• It supports access to elements via xpath elements, table IDs, text (regex-matching!) and URLs
• It provides a Swiss army knife of user-interaction options, giving you flexibility over how you wish to simulate end-user browsing

You'll need to download the Selenium Remote Control and have it running in the background for the module to work.

It may not be a good option if your page load times are unpredictable.