scala

问题 I've found in Akka docs: When using future callbacks, such as onComplete, onSuccess, and onFailure, inside actors you need to carefully avoid closing over the containing actor’s reference, i.e. do not call methods or access mutable state on the enclosing actor from within the callback. So does it mean that i should always use future pipeTo self and then call some functions? Or i can still use callbacks with method, then how should i avoid concurrency bugs? 回答1: It means this: class

问题 I have two rdd one rdd have just one column other have two columns to join the two RDD on key's I have add dummy value which is 0 , is there any other efficient way of doing this using join ? val lines = sc.textFile("ml-100k/u.data") val movienamesfile = sc.textFile("Cml-100k/u.item") val moviesid = lines.map(x => x.split("\t")).map(x => (x(1),0)) val test = moviesid.map(x => x._1) val movienames = movienamesfile.map(x => x.split("\\|")).map(x => (x(0),x(1))) val shit = movienames.join

问题 I have two rdd one rdd have just one column other have two columns to join the two RDD on key's I have add dummy value which is 0 , is there any other efficient way of doing this using join ? val lines = sc.textFile("ml-100k/u.data") val movienamesfile = sc.textFile("Cml-100k/u.item") val moviesid = lines.map(x => x.split("\t")).map(x => (x(1),0)) val test = moviesid.map(x => x._1) val movienames = movienamesfile.map(x => x.split("\\|")).map(x => (x(0),x(1))) val shit = movienames.join

问题 I am playing around with collections in Scala and found that mutable collections are defined as invariant and immutable collections are defined as covariant. What is the relation between variance and mutability / immutability in Scala? class Array[T] class List[+T] 回答1: I had found an easy explanation in SIA. Following is straight from there. Mutable objects need to be invariant A type parameter is invariant when it’s neither covariant nor contravariant. All Scala mutable collection classes

问题 I am playing around with collections in Scala and found that mutable collections are defined as invariant and immutable collections are defined as covariant. What is the relation between variance and mutability / immutability in Scala? class Array[T] class List[+T] 回答1: I had found an easy explanation in SIA. Following is straight from there. Mutable objects need to be invariant A type parameter is invariant when it’s neither covariant nor contravariant. All Scala mutable collection classes

问题 I want to split a string delimited by commas and use the result as either a Seq or a Set : def splitByComma(commaDelimited: String): Array[String] = commaDelimited.trim.split(',') def splitByCommaAsSet(commaDelimited: String): Set[String] = splitByComma(commaDelimited).toSet def splitByCommaAsSeq(commaDelimited: String): Seq[String] = splitByComma(commaDelimited).toSeq val foods = "sushi,taco,burrito" val foodSet = splitByCommaAsSet(foods) // foodSet: scala.collection.immutable.Set[String] =

问题 I want to split a string delimited by commas and use the result as either a Seq or a Set : def splitByComma(commaDelimited: String): Array[String] = commaDelimited.trim.split(',') def splitByCommaAsSet(commaDelimited: String): Set[String] = splitByComma(commaDelimited).toSet def splitByCommaAsSeq(commaDelimited: String): Seq[String] = splitByComma(commaDelimited).toSeq val foods = "sushi,taco,burrito" val foodSet = splitByCommaAsSet(foods) // foodSet: scala.collection.immutable.Set[String] =

问题 Spark can use Hadoop S3A file system org.apache.hadoop.fs.s3a.S3AFileSystem . By adding the following into the conf/spark-defaults.conf , I can get spark-shell to log to the S3 bucket: spark.jars.packages net.java.dev.jets3t:jets3t:0.9.0,com.google.guava:guava:16.0.1,com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.3 spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem spark.eventLog.enabled true spark.eventLog.dir s3a://spark-logs-test/ spark.history.fs

问题 If I have these two case classes: case class Address(street : String, zip : Int) case class Person(name : String, address : Address) and an instance: val person = Person("Jane", Address("street address", 12345)) Is there a way in shapeless to automatically convert person to an extensible record? I am interested in both shallow and deep conversions. The shallow copy would be something like: 'name ->> "Jane" :: 'address ->> Address("street address", 12345) :: HNil In the deep conversion, the

问题 I have CSV files with comments that give column names, where the columns change throughout the file: #c1,c2,c3 a,b,c d,e,f #c4,c5 g,h i,j I want to provide a way to iterate over (only) the data rows of the file as Maps of column names to values (all Strings). So the above would become: Map(c1 -> a, c2 -> b, c3 -> c) Map(c1 -> d, c2 -> e, c3 -> f) Map(c4 -> g, c5 -> h) Map(c4 -> i, c5 -> j) The files are very large, so reading everything into memory is not an option. Right now I have an