How can Perl's WWW::Mechanize expand HTML pages that add to themselves with JavaScript?

Question

As mentioned in a previous question, I\'m coding a crawler for the QuakeLive website.
I\'ve been using WWW::Mechanize to get the web content and this worked fine for a

Answer 1

The problem is that mechanize mimics the networking layer of the browser but not the rendering or javascript execution layer.

Many folks use the web browser control provided by Microsoft. This is a full instance of IE in a control that you can host in a WinForm, WPF or plain old Console app. It allows you to, among other things, load the web page and run javascript as well as send and receive javascript commands.

Here's a reasonable intro into how to host a browser control: http://www.switchonthecode.com/tutorials/csharp-snippet-tutorial-the-web-browser-control