Spark-Cassandra Connector : Failed to open native connection to Cassandra

Question

I am new to Spark and Cassandra. On trying to submit a spark job, I am getting an error while connecting to Cassandra.

Details:

Versions:

Spa         


        

Answer 1

I struggled with this issue overnight, and finally got a combination that works. I am writing it down for those who may run into similar issue.

First of all, this is a version issue cassandra-driver-core's dependency. But to track down the exact combination that works takes me quite a bit time.

Secondly, this is the combination that works for me.

1. Spark 1.6.2 with Hadoop 2.6, cassandra 2.1.5 (Ubuntu 14.04, Java 1.8),
2. In built.sbt (sbt assembly, scalaVersion := "2.10.5"), use

"com.datastax.spark" %% "spark-cassandra-connector" % "1.4.0", "com.datastax.cassandra" % "cassandra-driver-core" % "2.1.5"

Thirdly, let me clarify my frustrations. With spark-cassandra-connector 1.5.0, I can run the assembly with spark-submit with --master "local[2]" on the same machine with remote cassandra connection without any problem. Any combination of connector 1.5.0, 1.6.0 with Cassandra 2.0, 2.1, 2.2, 3,4 works well. But if I try to submit the job to a cluster from the same machine (NodeManager) with --master yarn --deploy-mode cluster, then I will always run into the problem: Failed to open native connection to Cassandra at {192.168.122.12}:9042

What is going on here? Any from DataStarX can take a look at this issue? I can only guess it has something to do with "cqlversion", which should match the version of Cassandra cluster.

Anybody know a better solution? [cassandra], [apache-spark]