scala-2.11

Spark 2.x - How to generate simple Explain/Execution Plan

自闭症网瘾萝莉.ら 提交于 2020-04-11 12:38:44
问题 I am hoping to generate an explain/execution plan in Spark 2.2 with some actions on a dataframe. The goal here is to ensure that partition pruning is occurring as expected before I kick off the job and consume cluster resources. I tried a Spark documentation search and a SO search here but couldn't find a syntax that worked for my situation. Here is a simple example that works as expected: scala> List(1, 2, 3, 4).toDF.explain == Physical Plan == LocalTableScan [value#42] Here's an example

How to get the annotations of a method in Scala 2.11

大憨熊 提交于 2020-01-13 07:08:09
问题 Let's assume a controller object like this: object Users extends Controller { ... @ApiOperation( httpMethod = "POST", nickname = "authenticate", value = "Authenticates an user", notes = "Returns the JSON Web Token to be used in any subsequent request", response = classOf[models.auth.api.Jwt]) def authenticate = SecuredAction[Users.type]("authenticate").async(parse.json) { implicit request => ... } ... } How do I get the annotation values of the authenticate method at runtime? I've tried this:

How to override an implicit value?

。_饼干妹妹 提交于 2020-01-01 05:15:06
问题 Suppose I have the code: class A(implicit s:String = "foo"){println(s)} object X { implicit val s1 = "hello" } object Y { import X._ // do something with X implicit val s2 = "hi" val a = new A } I get the error: <console>:14: error: ambiguous implicit values: both value s2 in object Y of type => String and value s1 in object X of type => String match expected type String val a = new A Is there any way I can tell Scala to use the value s2 in Y ? (if I rename s2 to s1 , it works as expected but

Access code file and line number from Scala macro?

让人想犯罪 __ 提交于 2019-12-23 17:20:57
问题 How can I access the name of the code file and line number in a Scala macro? I looked at SIP-19 and it says it can be easily implemented using macros... EDIT: To clarify, I want the code file and line number of the caller. I already have a debug macro and I want to modify it to print the line number and file name of whoever calls debug 回答1: You want c.macroApplication.pos , where c is for Context . c.enclosingPosition finds the nearest macro on the stack that has a position. (See the other

Scala instantiate objects from String classname

馋奶兔 提交于 2019-12-21 17:24:00
问题 I have a trait, Action, that many different classes, ${whatever}Action, extend. I'd like to make the class that is in charge of instantiating these Action objects dynamic in the sense that it will not know which of the extending objects it will be building until it is passed a String with the name. I want it to take the name of the class, then build the object based on that String. I'm having trouble finding a concise/recent answer regarding this simple bit of reflection. I was hoping to get

How to fix the Dropping Close since the SSL connection is already closing error in spray

三世轮回 提交于 2019-12-21 03:09:29
问题 I’m making a call to an API, but most of the time I keep getting an error: “ Dropping Close since the SSL connection is already closing ” and “ Premature connection close (the server doesn't appear to support request pipelining) .” Like 90% of the time I get that error, meaning: on very rare occasions the query does return the data it supposed to. To make sure this wasn’t the API’s server issue, I replicate the same query using Node.js (Express and Request libs) and it works every time. It

Why doesn't the Def.inputTask macro work in Scala 2.11.1?

这一生的挚爱 提交于 2019-12-19 05:03:19
问题 I'm using Scala 2.11.1 and sbt 0.13.5. I have an sbt plugin that contains a helper function to create input tasks as follows (the implementation is stripped away as it's irrelevant to the problem): def register(name: String, description: String): Def.Setting[InputTask[Unit]] = { InputKey[Unit](name, description) <<= Def.inputTask { println("test") } } This function compiles and works just fine in Scala 2.10.4, however once I switch to 2.11.1 it fails with the following error: can't expand

Scala macro to print code?

孤者浪人 提交于 2019-12-18 05:01:49
问题 I want to do something like this: def assuming[A](condition: => Boolean)(f: => A): A = { require(condition, /* print source-code of condition */) f } Sample usage: def fib(n: Int) = n match { // yes, yes, I know this is not efficient case 0 => 0 case 1 => 1 case i => assuming(i > 0) { fib(i-1) + fib(i-2) } } Now, for example, if you call fib(-20) , I want it to throw an exception with a message like Assertion failed: -20 > 0 or Assertation failed: i > 0 回答1: Dude, isn't an assert macro one of

Scala macro to print code?

不想你离开。 提交于 2019-12-18 05:01:06
问题 I want to do something like this: def assuming[A](condition: => Boolean)(f: => A): A = { require(condition, /* print source-code of condition */) f } Sample usage: def fib(n: Int) = n match { // yes, yes, I know this is not efficient case 0 => 0 case 1 => 1 case i => assuming(i > 0) { fib(i-1) + fib(i-2) } } Now, for example, if you call fib(-20) , I want it to throw an exception with a message like Assertion failed: -20 > 0 or Assertation failed: i > 0 回答1: Dude, isn't an assert macro one of

Dynamically compiling scala class files at runtime in Scala 2.11

北战南征 提交于 2019-12-17 10:42:48
问题 I have the following code that works in Scala 2.10 to compile external classes at runtime in Scala /** * Compile scala files and keep them loaded in memory * @param classDir Directory storing the generated scala files * @throws IOException if there is problem reading the source files * @return Classloader that contains the compiled external classes */ @throws[IOException] def compileFiles(classDir: String): AbstractFileClassLoader = { val files = recursiveListFiles(new File(classDir)) .filter