Puppeteer - infinite scrolling situation

问题

I wanted to keep scrolling down, until all the elements with a particular classname are loaded in a dynamic HTML environment.

This is the code i used:

while ((await page.$$('.xj7')).length < counter) {
 await page.evaluate(() =>
  window.scrollBy(0, window.innerHeight));
}

The problem is that after it loads all the elements, it doesn't stop scrolling. I don't know why is that, as it should exit the while loop.

When i terminate the application, i get this error:

(node:5708) UnhandledPromiseRejectionWarning: Error: Protocol error (Runtime.cal
lFunctionOn): Session closed. Most likely the page has been closed.
    at CDPSession.send (C:\node_modules\pupp
eteer\lib\Connection.js:187:29)
    at ExecutionContext.evaluateHandle (C:\node_modules\puppeteer\lib\ExecutionContext.js:73:75)
    at ExecutionContext.evaluate (C:\node_modules\puppeteer\lib\ExecutionContext.js:46:31)
    at Frame.evaluate (C:\node_modules\puppeteer\lib\FrameManager.js:326:20)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
(node:5708) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This
error originated either by throwing inside of an async function without a catch
block, or by rejecting a promise which was not handled with .catch(). (rejection
 id: 4)
(node:5708) [DEP0018] DeprecationWarning: Unhandled promise rejections are depre
cated. In the future, promise rejections that are not handled will terminate the
 Node.js process with a non-zero exit code.

As you can see it mentions unhandled promise rejection. Maybe it has to do with the new async function i introduced inside the while loop?

EDIT: I decided to test the value of the page.$$('.v1Nh3.kIKUG._bz0w')).length

I wrote this test code:

while ((await page.$$('.xj7')).length < counter) {
 const read = (await page.$$('.xj7')).length;
 console.log(read);
 await page.evaluate(() =>
  window.scrollBy(0, window.innerHeight));
}

The values increased from 12, to 24, then they got stuck at 30! Then when the page has loaded all the elements, the value dropped to 28!

That's why it stucks to this loop. I don't understand while the page has hundreds of these elements, it ranged to these values.

来源：https://stackoverflow.com/questions/50709049/puppeteer-infinite-scrolling-situation

标签

javascript

node.js

async-await

Puppeteer