apache-spark-dataset

问题 This article claims that a DataFrame in Spark is equivalent to a Dataset[Row] , but this blog post shows that a DataFrame has a schema. Take the example in the blog post of converting an RDD to a DataFrame : if DataFrame were the same thing as Dataset[Row] , then converting an RDD to a DataFrame should be as simple val rddToDF = rdd.map(value => Row(value)) But instead it shows that it's this val rddStringToRowRDD = rdd.map(value => Row(value)) val dfschema = StructType(Array(StructField(

问题 I am using spark-sql-2.4.1v with java8. Have to do a complex calculation using group by on various conditions using java api i.e. using MapFunction and ReduceFunction. Scenario : Have source data given sample as below +--------+--------------+-----------+-------------+---------+------+ | country|generated_date|industry_id|industry_name| revenue| state| +--------+--------------+-----------+-------------+---------+------+ |Country1| 2020-03-01| Indus_1| Indus_1_Name| 12789979|State1| |Country1|

问题 I am using spark-sql-2.4.1v with java8. Have to do a complex calculation using group by on various conditions using java api i.e. using MapFunction and ReduceFunction. Scenario : Have source data given sample as below +--------+--------------+-----------+-------------+---------+------+ | country|generated_date|industry_id|industry_name| revenue| state| +--------+--------------+-----------+-------------+---------+------+ |Country1| 2020-03-01| Indus_1| Indus_1_Name| 12789979|State1| |Country1|

问题 I had been reading about spark predicates pushdown and partition pruning to understand the amount of data read. I had the following doubts related to the same Suppose I have a dataset with columns (Year: Int, SchoolName: String, StudentId: Int, SubjectEnrolled: String) of which the data stored on disk is partitioned by Year and SchoolName and stored in parquet format at say azure data lake storage. 1) If I issue a read spark.read(container).filter(Year=2019, SchoolName="XYZ"): Will Partition

问题 I have a bunch of columns, sample like my data displayed as show below. I need to check the columns for errors and will have to generate two output files. I'm using Apache Spark 2.0 and I would like to do this in a efficient way. Schema Details --------------- EMPID - (NUMBER) ENAME - (STRING,SIZE(50)) GENDER - (STRING,SIZE(1)) Data ---- EMPID,ENAME,GENDER 1001,RIO,M 1010,RICK,MM 1015,123MYA,F My excepected output files should be as shown below: 1. EMPID,ENAME,GENDER 1001,RIO,M 1010,RICK,NULL

问题 I know there are many questions on the same but none really answers my question. I have scenario data. val data_codes = Seq("con_dist_1","con_dist_2","con_dist_3","con_dist_4","con_dist_5") val codes = data_codes.toDF("item_code") val partitioned_codes = codes.repartition($"item_code") println( "getNumPartitions : " + partitioned_codes.rdd.getNumPartitions); Output : getNumPartitions : 200 it suppose to give 5 right why it is giving 200 ? where am i doing wrong and how to fix this? 回答1:

问题 I know there are many questions on the same but none really answers my question. I have scenario data. val data_codes = Seq("con_dist_1","con_dist_2","con_dist_3","con_dist_4","con_dist_5") val codes = data_codes.toDF("item_code") val partitioned_codes = codes.repartition($"item_code") println( "getNumPartitions : " + partitioned_codes.rdd.getNumPartitions); Output : getNumPartitions : 200 it suppose to give 5 right why it is giving 200 ? where am i doing wrong and how to fix this? 回答1:

问题 I know there are many questions on the same but none really answers my question. I have scenario data. val data_codes = Seq("con_dist_1","con_dist_2","con_dist_3","con_dist_4","con_dist_5") val codes = data_codes.toDF("item_code") val partitioned_codes = codes.repartition($"item_code") println( "getNumPartitions : " + partitioned_codes.rdd.getNumPartitions); Output : getNumPartitions : 200 it suppose to give 5 right why it is giving 200 ? where am i doing wrong and how to fix this? 回答1:

问题 I'm trying to figure how I can update some rows based on another another row. For example, I have some data like Id | useraname | ratings | city -------------------------------- 1, philip, 2.0, montreal, ... 2, john, 4.0, montreal, ... 3, charles, 2.0, texas, ... I want to update the users in the same city to the same groupId (either 1 or 2) Id | useraname | ratings | city -------------------------------- 1, philip, 2.0, montreal, ... 1, john, 4.0, montreal, ... 3, charles, 2.0, texas, ...

问题 I'm trying to figure how I can update some rows based on another another row. For example, I have some data like Id | useraname | ratings | city -------------------------------- 1, philip, 2.0, montreal, ... 2, john, 4.0, montreal, ... 3, charles, 2.0, texas, ... I want to update the users in the same city to the same groupId (either 1 or 2) Id | useraname | ratings | city -------------------------------- 1, philip, 2.0, montreal, ... 1, john, 4.0, montreal, ... 3, charles, 2.0, texas, ...