spark-cassandra-connector

I have a simple Spark job reading 500m rows from a 5 node Cassandra cluster that always runs 6 tasks, which is causing write issues due to the size of each task. I have tried adjusting the input_split_size, which seems to have no effect. At the moment I am forced to repartition the table scan, which is not ideal as it's expensive. Having read a few posts I tried to increase the num-executors in my launch script (below), although this had no effect. If there is no way to set the number of tasks on a Cassandra table scan, that's fine I'll make do.. but I have this constant niggling feeling that

We are using Kafka,Spark Streaming and loading data to Cassandra Need to implement a security layer between nodes running kafka and nodes running spark. Any guidance on how to implement SSL between kafka and spark nodes ? Thanks Sreeni 来源： https://stackoverflow.com/questions/37743490/ssl-between-kafka-and-spark

I am trying to use Apache Spark to process my large (~230k entries) cassandra dataset, but I am constantly running into different kinds of errors. However I can successfully run applications when running on a dataset ~200 entries. I have a spark setup of 3 nodes with 1 master and 2 workers, and the 2 workers also have a cassandra cluster installed with data indexed with a replication factor of 2. My 2 spark workers show 2.4 and 2.8 GB memory on the web interface and I set spark.executor.memory to 2409 when running an application, to get a combined memory of 4.7 GB. Here is my WebUI Homepage

I have created a simple Java application that uses Apache Spark to retrieve data from Cassandra, do some transformation on it and save it in another Cassandra table. I am using Apache Spark 1.4.1 configured in a standalone cluster mode with a single master and slave, located on my machine. DataFrame customers = sqlContext.cassandraSql("SELECT email, first_name, last_name FROM customer " + "WHERE CAST(store_id as string) = '" + storeId + "'"); DataFrame customersWhoOrderedTheProduct = sqlContext.cassandraSql("SELECT email FROM customer_bought_product " + "WHERE CAST(store_id as string) = '" +

I have the following piece of code in Spark: rdd .map(processFunction(_)) .saveToCassandra("keyspace", "tableName") Where def processFunction(src: String): Seq[Any] = src match { case "a" => List(A("a", 123112, "b"), A("b", 142342, "c")) case "b" => List(B("d", 12312, "e", "f"), B("g", 12312, "h", "i")) } Where: case class A(entity: String, time: Long, value: String) case class B(entity: String, time: Long, value1: String, value2: String) saveToCassandra expects a collection of objects and using Seq[Any] as the return type to contain both Seq[A] and Seq[B] breaks saveToCassandra with the

I'm doing a benchmark on Cassandra's Reading performance. In the test-setup step I created a cluster with 1 / 2 / 4 ec2-instances and data nodes. I wrote 1 table with 100 million of entries (~3 GB csv-file). Then I launch a Spark application which reads the data into a RDD using the spark-cassandra-connector. However, I thought the behavior should be the following: The more instances Cassandra (same instance amount on Spark) uses, the faster the reads! With the writes everything seems to be correct (~2-times faster if cluster 2-times larger). But: In my benchmark the read is always faster with

I am trying to save the streaming data to cassandra from Kafka. I am able to read and parse the data but when I call below lines to save the data i am getting a Task not Serializable Exception. My class is extending serializable but not sure why i am seeing this error, didn't get much help ever after googling for 3 hours, can some body give any pointers ? val collection = sc.parallelize(Seq((obj.id, obj.data))) collection.saveToCassandra("testKS", "testTable ", SomeColumns("id", "data"))` import org.apache.spark.SparkConf import org.apache.spark.SparkContext import org.apache.spark.sql