Node.js Twitter API cursors

Question

I'm using npm-twit to get followers of a specific account.

The Twitter API returns up to 5000 results from a single GET request.

If the user I'm querying has over 5000 followers a "next_cursor" value is returned with the data.

To get the next 5000 results, I need to re-run the GET function, passing it the "next_cursor" value as an argument. I just can't seem to work out how to do it.

I was thinking a while loop, but I can't reset the global variable, I think because of scope:

var cursor = -1

while ( cursor != 0 ) { 

  T.get('followers/ids', { screen_name: 'twitter' },  function (err, data, response) {

  // Do stuff here to write data to a file

  cursor = data["next_cursor"];

  })

}

Obviously I'm not a JS genius, so any help would be much appreciated.

Answer 1

The issue you are having is due to Node.js being asynchronous.

T.get('followers/ids', { screen_name: 'twitter' },  function getData(err, data, response) {

  // Do stuff here to write data to a file

  if(data['next_cursor'] > 0) T.get('followers/ids', { screen_name: 'twitter', next_cursor: data['next_cursor'] }, getData);

  })

}

Please note:

1. I gave a name to the internal callback function. That is so that we can recursively call it from the inside.
2. The loop is replaced with a recursive callback.
3. If there is a next_cursor data, then we call T.get using the same function getData.

Be aware that Do stuff here code will be executed many times (as many as there are next cursors). Since it is recursive callback - the order is guaranteed.

---

If you do not like the idea of recursive callbacks, you can avoid it by:

1. Finding out beforehand all the next_cursor's if possible, and generate requests using for loop.
2. Alternatively, use asynchronous-helper modules like Async (though for learning purposes, I would avoid modules unless you are fluent in the concept already).

Answer 2

Consider testing with some 5K+ account.

    const T = new Twit(tokens)

    function getFollowers (screenName, followers = [], cur = -1) {
      return new Promise((resolve, reject) => {
        T.get('followers/ids', { screen_name: screenName, cursor: cur, count: 5000 }, (err, data, response) => {
          if (err) {
            cur = -1
            reject(err)
          } else {
            cur = data.next_cursor
            followers.push(data.ids)
            if (cur > 0) {
              return resolve(getFollowers(screenName, followers, cur))
            } else {
              return resolve([].concat(...followers))
            }
          }
        })
      })
    }

    async function getXaqron () {
      let result = await getFollowers('xaqron')
      return result
    }

 console.log(getXaqron().catch((err) => {
  console.log(err) // Rate limit exceeded
}))

Answer 3

Struggled with this one.. Everything seemed to work, but data['next_cursor'] didn't change, EVER!

Code should be like this:

T.get('followers/ids', { screen_name: 'twitter' },  function getData(err, data, response) {

  // Do stuff here to write data to a file

  if(data['next_cursor'] > 0) T.get('followers/ids', { screen_name: 'twitter', cursor: data['next_cursor'] }, getData);

  })

}

Parameter for Twit isn't "next_cursor", it's just "cursor" ;)