apache-spark-dataset

I am working on the spark code where I have to save multiple column values as a object format and save the result to mongodb Given Dataset |---|-----|------|----------| |A |A_SRC|Past_A|Past_A_SRC| |---|-----|------|----------| |a1 | s1 | a2 | s2 | What I Have tried val ds1 = Seq(("1", "2", "3","4")).toDF("a", "src", "p_a","p_src") val recordCol = functions.to_json(Seq($"a", $"src", $"p_a",$"p_src"),struct("a", "src", "p_a","p_src")) as "A" ds1.select(recordCol).show(truncate = false) gives me result like +-----------------------------------------+ |A | +---------------------------------------

SparkSession.createDataset() only allows List, RDD, or Seq - but it doesn't support JavaPairRDD . So if I have a JavaPairRDD<String, User> that I want to create a Dataset from, would a viable workround for the SparkSession.createDataset() limitation to create a wrapper UserMap class that contains two fields: String and User . Then do spark.createDataset(userMap, Encoders.bean(UserMap.class)); ? If you can convert the JavaPairRDD to List<Tuple2<K, V>> then you can use createDataset method which takes List. See below sample code. JavaPairRDD<String, User> pairRDD = ...; Dataset<Row> df = spark

I have 2 parquet part files part-00043-0bfd7e28-6469-4849-8692-e625c25485e2-c000.snappy.parquet (is part file from 2017 Nov 14th run ) and part-00199-64714828-8a9e-4ae1-8735-c5102c0a834d-c000.snappy.parquet (is part file from 2017 Nov 16th run ) and have both having same schema (which I verified by printing schema). My problem is that I have, say 10 columns which is coming properly if I read this 2 files separately using Spark. But if I put this file is folder are try to read together, total count is coming correct (sum of rows from 2 files) but from 2nd file most of the columns are null. Only

In the following snippet, the tryParquet function tries to load a Dataset from a Parquet file if it exists. If not, it computes, persists and returns back the Dataset plan which was provided: import scala.util.{Try, Success, Failure} import org.apache.spark.sql.SparkSession import org.apache.spark.sql.Dataset sealed trait CustomRow case class MyRow( id: Int, name: String ) extends CustomRow val ds: Dataset[MyRow] = Seq((1, "foo"), (2, "bar"), (3, "baz")).toDF("id", "name").as[MyRow] def tryParquet[T <: CustomRow](session: SparkSession, path: String, target: Dataset[T]): Dataset[T] = Try

Trying to understand how Hive partitions relate to Spark partitions, culminating in a question about joins. I have 2 external Hive tables; both backed by S3 buckets and partitioned by date ; so in each bucket there are keys with name format date=<yyyy-MM-dd>/<filename> . Question 1: If I read this data into Spark: val table1 = spark.table("table1").as[Table1Row] val table2 = spark.table("table2").as[Table2Row] then how many partitions are the resultant datasets going to have respectively? Partitions equal to the number of objects in S3? Question 2 : Suppose the two row types have the following

I'm writing a Spark application using version 2.1.1. The following code got the error when calling a method with LocalDate parameter? Exception in thread "main" java.lang.UnsupportedOperationException: No Encoder found for java.time.LocalDate - field (class: "java.time.LocalDate", name: "_2") - root class: "scala.Tuple2" at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$serializerFor(ScalaReflection.scala:602) at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:596) at org.apache.spark.sql.catalyst

I want to assign a unique Id to my dataset rows. I know that there are two implementation options: First option: import org.apache.spark.sql.expressions.Window; ds.withColumn("id",row_number().over(Window.orderBy("a column"))) Second option: df.withColumn("id", monotonically_increasing_id()) The second option is not sequential ID and it doesn't really matter. I'm trying to figure out is if there are any performance issues of those implementation. That is, if one of this option is very slow compared to the other. Something more meaningful that: "monotonically_increasing_id is very fast over row