spark-dataframe

I want to load cassandra table to a datafram in spark, I have followed the sample programes below (found in this answer ), but I am getting an execption mentioned below, I have tried to load the table to RDD first then convert it to Datafrme, loading the RDD is successful, but when I try to convert it to a dataframe I am getting the same execption faced in the first methdology, any suggestions ? I am using Spark 2.0.0, Cassandra 3.7, and Java 8. public class SparkCassandraDatasetApplication { public static void main(String[] args) { SparkSession spark = SparkSession .builder() .appName(

I am working on spark dataframes and I need to do a group by of a column and convert the column values of grouped rows into an array of elements as new column. Example : Input: employee | Address ------------------ Micheal | NY Micheal | NJ Output: employee | Address ------------------ Micheal | (NY,NJ) Any help is highly appreciated.! Here is an alternate solution Where I have converted the dataframe to an rdd for the transformations and converted it back a dataFrame using sqlContext.createDataFrame() Sample.json {"employee":"Michale","Address":"NY"} {"employee":"Michale","Address":"NJ"} {

I use spark1.6. I tried to broadcast a RDD and am not sure how to access the broadcasted variable in the data frames? I have two dataframes employee & department. Employee Dataframe ------------------- Emp Id | Emp Name | Emp_Age ------------------ 1 | john | 25 2 | David | 35 Department Dataframe -------------------- Dept Id | Dept Name | Emp Id ----------------------------- 1 | Admin | 1 2 | HR | 2 import scala.collection.Map val df_emp = hiveContext.sql("select * from emp") val df_dept = hiveContext.sql("select * from dept") val rdd = df_emp.rdd.map(row => (row.getInt(0),row.getString(1)))

My input dataset is about 150G. I am setting --conf spark.cores.max=100 --conf spark.executor.instances=20 --conf spark.executor.memory=8G --conf spark.executor.cores=5 --conf spark.driver.memory=4G but since data is not evenly distributed across executors, I kept getting Container killed by YARN for exceeding memory limits. 9.0 GB of 9 GB physical memory used here are my questions: 1. Did I not set up enough memory in the first place? I think 20 * 8G > 150G, but it's hard to make perfect distribution, so some executors will suffer 2. I think about repartition the input dataFrame, so how can I

I've a .gz file available on a Web server that I want to consume in a streaming manner and insert the data into Couchbase. The .gz file has only one file in it, which in turn contains one JSON object per line. Since Spark doesn't have a HTTP receiver, I wrote one myself (shown below). I'm using Couchbase Spark connector to do the insertion. However, when running, the job is not actually inserting anything. I've a suspicion that it is due to my inexperience with Spark and not knowing how to start and await termination. As you can see below, there are 2 places such calls can be made. Receiver :

In Spark's documentation, Aggregator: abstract class Aggregator[-IN, BUF, OUT] extends Serializable A base class for user-defined aggregations, which can be used in Dataset operations to take all of the elements of a group and reduce them to a single value. UserDefinedAggregateFunction is: abstract class UserDefinedAggregateFunction extends Serializable The base class for implementing user-defined aggregate functions (UDAF). According to Dataset Aggregator - Databricks , “an Aggregator is similar to a UDAF, but the interface is expressed in terms of JVM objects instead of as a Row .” It seems

In one of my spark job (2.0 on EMR 5.0.0) where I had about 5GB of data that was crossed joined with 30 rows(data size few MBs). I further needed to group by it. What I noticed that I was taking lot of time (Approximately 4 hours with one m3.xlarge master and six m3.2xlarge core nodes). In total time 2 hour was taken by processing and another 2 hour was taken to write data to s3. The time taken was not very impressive to me. I tried searching over net and found this link that says groupBy leads lot of shuffling. It also suggests that for avoiding lot of shuffling ReduceByKey should be used