scala

问题 I look for a way to retrieve the first elements of a DStream created as: val dstream = ssc.textFileStream(args(1)).map(x => x.split(",").map(_.toDouble)) Unfortunately, there is no take function (as on RDD) on a dstream // dstream.take(2) !!! Could someone has any idea on how to do it ?! thanks 回答1: You can use transform method in the DStream object then take n elements of the input RDD and save it to a list, then filter the original RDD to be contained in this list. This will return a new

问题 I am currently experiencing a weird exception when testing my Scala - Spring Boot application with JUnit Vintage. java.lang.IncompatibleClassChangeError: org.myapp.SampleWebApplication and org.myapp.SampleWebApplication$delayedInit$body disagree on InnerClasses attribute at java.lang.Class.getDeclaringClass0(Native Method) at java.lang.Class.getDeclaringClass(Class.java:1235) at java.lang.Class.getEnclosingClass(Class.java:1277) at java.lang.Class.getSimpleBinaryName(Class.java:1443) at java

问题 I am currently experiencing a weird exception when testing my Scala - Spring Boot application with JUnit Vintage. java.lang.IncompatibleClassChangeError: org.myapp.SampleWebApplication and org.myapp.SampleWebApplication$delayedInit$body disagree on InnerClasses attribute at java.lang.Class.getDeclaringClass0(Native Method) at java.lang.Class.getDeclaringClass(Class.java:1235) at java.lang.Class.getEnclosingClass(Class.java:1277) at java.lang.Class.getSimpleBinaryName(Class.java:1443) at java

问题 I am currently experiencing a weird exception when testing my Scala - Spring Boot application with JUnit Vintage. java.lang.IncompatibleClassChangeError: org.myapp.SampleWebApplication and org.myapp.SampleWebApplication$delayedInit$body disagree on InnerClasses attribute at java.lang.Class.getDeclaringClass0(Native Method) at java.lang.Class.getDeclaringClass(Class.java:1235) at java.lang.Class.getEnclosingClass(Class.java:1277) at java.lang.Class.getSimpleBinaryName(Class.java:1443) at java

问题 I want to create a Flow that have 4 main steps (FlowShapes), and after the first and the second I want to have partition that will decide if there is a reason to go to the next, and if not to sink it so the stream will pick it up later and start from beginning, but im not sure this is the way, cause I just used Sink.ignore, it looks like this: def mainFlow: Flow[MyGraphElement, MyGraphElement, NotUsed] = Flow.fromGraph(GraphDSL.create() { implicit builder => // FlowShape's // those flow

问题 I want to create a Flow that have 4 main steps (FlowShapes), and after the first and the second I want to have partition that will decide if there is a reason to go to the next, and if not to sink it so the stream will pick it up later and start from beginning, but im not sure this is the way, cause I just used Sink.ignore, it looks like this: def mainFlow: Flow[MyGraphElement, MyGraphElement, NotUsed] = Flow.fromGraph(GraphDSL.create() { implicit builder => // FlowShape's // those flow

问题 Spark API RelationalGroupedDataset has a function agg : @scala.annotation.varargs def agg(expr: Column, exprs: Column*): DataFrame = { toDF((expr +: exprs).map { case typed: TypedColumn[_, _] => typed.withInputType(df.exprEnc, df.logicalPlan.output).expr case c => c.expr }) } Why does it take two separate arguments? Why can't it take just exprs: Column* ? Has someone an implicit function that takes one argument? 回答1: This is to make sure that you specify at least one argument. Pure varargs

问题 Spark API RelationalGroupedDataset has a function agg : @scala.annotation.varargs def agg(expr: Column, exprs: Column*): DataFrame = { toDF((expr +: exprs).map { case typed: TypedColumn[_, _] => typed.withInputType(df.exprEnc, df.logicalPlan.output).expr case c => c.expr }) } Why does it take two separate arguments? Why can't it take just exprs: Column* ? Has someone an implicit function that takes one argument? 回答1: This is to make sure that you specify at least one argument. Pure varargs

问题 It seems like StructType preserves order, so two StructType containing same StructField s are not considered equivalent. For example: val st1 = StructType( StructField("ii",StringType,true) :: StructField("i",StringType,true) :: Nil) val st2 = StructType( StructField("i",StringType,true) :: StructField("ii",StringType,true) :: Nil) println(st1 == st2) returns false even though they both have StructField("i",StringType,true) and StructField("ii",StringType,true) , just in different order. I

问题 It seems like StructType preserves order, so two StructType containing same StructField s are not considered equivalent. For example: val st1 = StructType( StructField("ii",StringType,true) :: StructField("i",StringType,true) :: Nil) val st2 = StructType( StructField("i",StringType,true) :: StructField("ii",StringType,true) :: Nil) println(st1 == st2) returns false even though they both have StructField("i",StringType,true) and StructField("ii",StringType,true) , just in different order. I