spark-cassandra-connector

New to the Spark environment (and fairly new to Maven) so I'm struggling with how to send the dependencies I need correctly. It looks like Spark 1.5.1 has a guava-14.0.1 dependency which it tries to use and the isPrimitive was added in 15+. What's the correct way to ensure my uber-jar wins? I've tried spark.executor.extraClassPath in my spark-defaults.conf to no avail. Duplicate to this [question]: Spark 1.5.1 + Scala 2.10 + Kafka + Cassandra = Java.lang.NoSuchMethodError: but for Maven essentially (don't have rep to comment yet) Stripped down my dependencies to this: <dependency> <groupId>com

I'm fairly new to Scala and am trying to build a Spark job. I've built ajob that contains the DataStax connector and assembled it into a fat jar. When I try to execute it it fails with a java.lang.NoSuchMethodError . I've cracked open the JAR and can see that the DataStax library is included. Am I missing something obvious? Is there a good tutorial to look at regarding this process? Thanks console $ spark-submit --class org.bobbrez.CasCountJob ./target/scala-2.11/bobbrez-spark-assembly-0.0.1.jar ks tn ... Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.ObjectRef.zero(

I have a problem when i use spark streaming to read from Cassandra. https://github.com/datastax/spark-cassandra-connector/blob/master/doc/8_streaming.md#reading-from-cassandra-from-the-streamingcontext As the link above, i use val rdd = ssc.cassandraTable("streaming_test", "key_value").select("key", "value").where("fu = ?", 3) to select the data from cassandra, but it seems that the spark streaming has just one query once but i want it continues to query using an interval 10 senconds. My code is as follow, wish for your response. Thanks! import org.apache.spark._ import org.apache.spark

I am using the Spark Cassandra connector. It take 5-6 minutes for fetch data from Cassandra table. In Spark I have seen many tasks and Executor in log. The reason might be that Spark divided the process in many tasks! Below is my code example : public static void main(String[] args) { SparkConf conf = new SparkConf(true).setMaster("local[4]") .setAppName("App_Name") .set("spark.cassandra.connection.host", "127.0.0.1"); JavaSparkContext sc = new JavaSparkContext(conf); JavaRDD<Demo_Bean> empRDD = javaFunctions(sc).cassandraTable("dev", "demo"); System.out.println("Row Count"+empRDD.count()); }

I am getting data from somewhere and inserting it into cassandra daily basis then I need to retrieve the data from cassandra for whole week and do some processing and insert result back onto cassandra . i have lot of records, each record executing most of the below operations. To do this I have written a program below its working fine but I get warning and according to API document should not use prepare statement multiple time its reducing performance. Please tell me how to avoid this to improve the performance OR suggest me any alternative approach to achieve this in scala. Here is some part

I'm always getting the following error.Can somebody help me please? Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security