scala

问题 While toying with a tail recursion example I noticed a small discrepancy between the results of a normal recursive call and a tail recursive call: scala> def fact(n: Int): Double = if(n < 1) 1 else n * fact(n - 1) fact: (n: Int)Double scala> fact(30) res31: Double = 2.6525285981219103E32 scala> @tailrec def fact(n: Int, acc: Double = 1): Double = if(n < 1) acc else fact(n - 1, n * acc) fact: (n: Int, acc: Double)Double scala> fact(30) res32: Double = 2.652528598121911E32 Just out of curiosity

问题 This question already has an answer here : Why does using cache on streaming Datasets fail with “AnalysisException: Queries with streaming sources must be executed with writeStream.start()”? (1 answer) Closed 13 days ago . I'm new with Apache Spark Structured Streaming. I'm trying to read some events from Event Hub (in XML format) and trying to create new Spark DF from the nested XML. Im using the code example described in https://github.com/databricks/spark-xml and in batch mode is running

问题 This question already has an answer here : Why does using cache on streaming Datasets fail with “AnalysisException: Queries with streaming sources must be executed with writeStream.start()”? (1 answer) Closed 13 days ago . I'm new with Apache Spark Structured Streaming. I'm trying to read some events from Event Hub (in XML format) and trying to create new Spark DF from the nested XML. Im using the code example described in https://github.com/databricks/spark-xml and in batch mode is running

问题 While trying to get year and week number of a range of dates spanning multiple years, I am getting into some issues with the start/end of the year. I understand the logic for weeknumber and the one of year when they run separately. However, when they are combined, in some cases they don't bring consistent results and I was wondering what is the best way in Spark to make sure that those scenarios are handled with a consistent year for the given weeknumber, For example, running: spark.sql(

问题 While trying to get year and week number of a range of dates spanning multiple years, I am getting into some issues with the start/end of the year. I understand the logic for weeknumber and the one of year when they run separately. However, when they are combined, in some cases they don't bring consistent results and I was wondering what is the best way in Spark to make sure that those scenarios are handled with a consistent year for the given weeknumber, For example, running: spark.sql(

问题 Hi I would like to know if using SBT is possible to cross compiling against different Scala version using different sources for some classes. To keep back compatibility basically but leverage also on the new language features. Thanks. 回答1: You can add additional source directories based on scala version by adding to the unmanagedSourceDirectories setting. Something like this: unmanagedSourceDirectories in Compile <+= (scalaVersion, sourceDirectory in Compile) { case (v, dir) if v startsWith

问题 Consider the following simple example involving Aux -pattern: sealed trait AdtBase abstract case class Foo(){ type T <: AdtBase } object Foo{ type Aux[TT] = Foo { type T = TT } } abstract case class Bar(){ type T <: AdtBase val foo: Foo.Aux[T] } object Bar { type Aux[TT] = Bar { type T = TT } def apply[TT <: AdtBase](f: Foo.Aux[TT]): Bar = new Bar() { override type T = TT override val foo: Foo.Aux[T] = f } } case class Baz(foo: Foo) def testBaz(baz: Baz) = Bar(baz.foo) //Compiles fine def

问题 I'm trying to understand databricks delta and thinking to do a POC using Kafka. Basically the plan is to consume data from Kafka and insert it to the databricks delta table. These are the steps that I did: Create a delta table on databricks. %sql CREATE TABLE hazriq_delta_trial2 ( value STRING ) USING delta LOCATION '/delta/hazriq_delta_trial2' Consume data from Kafka. import org.apache.spark.sql.types._ val kafkaBrokers = "broker1:port,broker2:port,broker3:port" val kafkaTopic = "kafkapoc"

问题 How to emulate following behavior in Scala? i.e. keep folding while some certain conditions on the accumulator are met. def foldLeftWhile[B](z: B, p: B => Boolean)(op: (B, A) => B): B For example scala> val seq = Seq(1, 2, 3, 4) seq: Seq[Int] = List(1, 2, 3, 4) scala> seq.foldLeftWhile(0, _ < 3) { (acc, e) => acc + e } res0: Int = 1 scala> seq.foldLeftWhile(0, _ < 7) { (acc, e) => acc + e } res1: Int = 6 UPDATES: Based on @Dima answer, I realized that my intention was a little bit side

问题 How to split an iterator into a prefix with duplicates and the rest ? For instance, def splitDupes(it: Iterator[Int]): (Iterator[Int], Iterator[Int]) = ??? val (xs, ys) = splitDupes(List(1, 1, 1, 2, 3, 4, 5).iterator) xs.toList // List(1, 1, 1) ys.toList // List(2, 3, 4, 5) val (xs, ys) = splitDupes(List(1, 2, 3, 4, 5).iterator) xs.toList // List(1) ys.toList // List(2, 3, 4, 5) val (xs, ys) = splitDupes(List(1, 1, 1, 1, 1).iterator) xs.toList // List(1, 1, 1, 1, 1) ys.toList // List() val