apache-spark-dataset

I need to iterate over data frame in specific order and apply some complex logic to calculate new column. In below example I'll be using simple expression where current value for s is multiplication of all previous values thus it may seem like this can be done using UDF or even analytic functions. However, in reality logic is much more complex. Below code does what is needed import org.apache.spark.sql.Row import org.apache.spark.sql.types._ import org.apache.spark.sql.catalyst.encoders.RowEncoder val q = """ select 10 x, 1 y union all select 10, 2 union all select 10, 3 union all select 20, 6

How can I get this method to compile. Strangely, sparks implicit are already imported. def loadDsFromHive[T <: Product](tableName: String, spark: SparkSession): Dataset[T] = { import spark.implicits._ spark.sql(s"SELECT * FROM $tableName").as[T] } This is the error: Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases. [error] spark.sql(s"SELECT * FROM $tableName").as[T] According to the source code for org.apache

I am working on a dataset which represents a stream of events (like fired as tracking events from a website). All the events have a timestamp. One use case we often have is trying to find the 1st non null value for a given field. So for example something like gets us most the way there: val eventsDf = spark.read.json(jsonEventsPath) case class ProjectedFields(visitId: String, userId: Int, timestamp: Long ... ) val projectedEventsDs = eventsDf.select( eventsDf("message.visit.id").alias("visitId"), eventsDf("message.property.user_id").alias("userId"), eventsDf("message.property.timestamp"), ...

I am struggling with the implementation in spark streaming. The messages from the kafka looks like this but with with more fields {"event":"sensordata", "source":"sensors", "payload": {"actual data as a json}} {"event":"databasedata", "mysql":"sensors", "payload": {"actual data as a json}} {"event":"eventApi", "source":"event1", "payload": {"actual data as a json}} {"event":"eventapi", "source":"event2", "payload": {"actual data as a json}} I am trying to read the messages from a Kafka topic (which has multiple schemas). I need to read each message and look for an event and source field and

Getting this null error in spark Dataset.filter Input CSV: name,age,stat abc,22,m xyz,,s Working code: case class Person(name: String, age: Long, stat: String) val peopleDS = spark.read.option("inferSchema","true") .option("header", "true").option("delimiter", ",") .csv("./people.csv").as[Person] peopleDS.show() peopleDS.createOrReplaceTempView("people") spark.sql("select * from people where age > 30").show() Failing code ( Adding following lines return error ): val filteredDS = peopleDS.filter(_.age > 30) filteredDS.show() Returns null error java.lang.RuntimeException: Null value appeared in