Using multiple page.open in one script

Question

My goal is open many pages(with a short delay) and save my data to a file.

But my code does not work.

var gamesList = [url1,url2,url3];
//gamesList i         


        

Answer 1

Your idea of opening multiple pages with recursion is correct, but you have some problems.

Exit

As you correctly noted, you have a problem with phantom.exit(). Since page.open() and setTimeout() are asynchronous, you only need to exit when you are done. When you call phantom.exit() at the end of the script, you're exiting before the first page is even loaded.

Simply remove that last phantom.exit(), because you already have another exit at the correct place.

Page context

page.evaluate() provides access to the DOM context (page context). The problem is that it is sandboxed. Inside of that callback you have no access to variables defined outside. You can explicitly pass variables in, but they have to be primitive objects which page is not. You simply have to access to page inside of page.evaluate(). You need to inject jQuery before calling page.evaluate().

Files

You're overwriting the file in every iteration by not changing the file name. Either you need to change the filename or use the appending mode 'a' instead of 'w'.

Then you don't need to open a stream when you simply want to write once. Change:

var file = fs.open('new_test.txt', "w");
file.write(html + '\n');
file.close();

to

fs.write('new_test.txt', html + '\n', 'a');

Recursive step

The recursive step with calling the next_page() function requires that you pass in the urls. Since urls is already a global variable and you change it in each iteration, you don't need to pass in the urls.

You also don't need to add a setTimeout(), because everything before inside of the page.open() callback was synchronous.

Fixed Script

//...
var urls = [/*....*/];

function handle_page(url){
    page.open(url, function(){
        //...
        page.injectJs('jquery.min.js');
        var html = page.evaluate(function(){
            // ...do stuff...
            return $('body').html();
        });
        //save to file
        fs.write('new_test.txt', html + '\n', 'a');

        console.log(html);

        next_page();
    });
}

function next_page(){
    var url = urls.shift();
    if(!url){
        phantom.exit(0);
    }
    handle_page(url);
}

next_page();