Applying update query on a record of 100,000 users on cassandra table using nodejs. Busy Connection issue

问题

I'm working with cassandra 3.x and node 10.13.0. I've data of 100,000 users in my working memory (in a map 'sortedRowMap' as in given code). I'm updating all of the record by iterating the map (having record of 100,000) users using for each. But it's throwing me BusyConnectionError. I'm wondering how could I get out from this.

Below is the implemented code of the aforementioned description.

var cassClient = new cassandra.Client({contactPoints: ['localhost'],pooling: {
    coreConnectionsPerHost: {
        [distance.local] : 2,
        [distance.remote] : 1
    },
}, keyspace: 'xyz',
    socketOptions: { readTimeout: 65000 }
});

rank = 0;

for (const [msisdn, totalearnings] of sortedRowMap) {

     let updateRankQuery = "UPDATE users SET weeklyrank = " + rank + " WHERE 
     msisdn = " + msisdn;

     cassClient.execute(updateRankQuery, function (error, result) {
            if(!error){
                rank++;
                console.log("updateQuery: " + updateRankQuery)

            }else{
                console.log("ERROR: " + error)
            }
        })

}

and its throwing me this error:

ERROR: NoHostAvailableError: All host(s) tried for query failed. First host tried, 127.0.0.1:9042: BusyConnectionError: All connections to host 127.0.0.1:9042 are busy, 2048 requests are in-flight on each connection. See innerErrors.

回答1:

You need to increase the number of in-flight requests per connection. Protocol version 3+ support up to 32k in-flight requests. You need to add corresponding option (maxRequestsPerConnection) into pooling object, like, this:

pooling: {
    coreConnectionsPerHost: {
        [distance.local] : 2,
        [distance.remote] : 1
    },
    maxRequestsPerConnection: 32768
}

But you can still hit this exception if all your requests are hitting the same nodes. In this case you'll need to throttle your submissions, or re-try them

Also see corresponding section in DataStax Development Guide.

来源：https://stackoverflow.com/questions/54301968/applying-update-query-on-a-record-of-100-000-users-on-cassandra-table-using-node