pureconfig

Spark 2.1 with Pureconfig 0.8 workaround for Maven

由 守給你的承諾、 提交于 2019-12-24 20:49:50
问题 The below solution taken from Spark not working with pureconfig seems to be the working solution for sbt but having a hard time figuring out a maven version for doing this. Trying to get pureconfig 0.8 working with spark 2.1 using spark-submit but still getting the pesky Exception in thread "main" java.lang.NoSuchMethodError: shapeless.Witness$.mkWitness(Ljava/lang/Object;)Lshapeless/Witness; error outside of IntelliJ. assemblyShadeRules in assembly := Seq( ShadeRule.rename("shapeless.**" ->

scala generic type hierarchy with different attribute name

由 寵の児 提交于 2019-12-23 05:06:52
问题 I use https://pureconfig.github.io/ to load configuration values. For example for each table in a database, I store (db: String, table: String) .However, I need to denote specific tables. Therefore, each one has a separate trait. I.e.: trait Thing trait ThingWithStuff extends Thing { def value:String } trait FooThing extends Thing{ def fooThing: ThingWithStuff } trait BarThing extends Thing{ def barThing: ThingWithStuff } They all have a different attribute name with the same type which in

Spark not working with pureconfig

由 前提是你 提交于 2019-11-29 14:57:27
I'm trying to use pureConfig and configFactory for my spark application configuration. here is my code: import pureconfig.{loadConfigOrThrow} object Source{ def apply(keyName: String, configArguments: Config): Source = { keyName.toLowerCase match { case "mysql" => val properties = loadConfigOrThrow[DBConnectionProperties](configArguments) new MysqlSource(None, properties) case "files" => val properties = loadConfigOrThrow[FilesSourceProperties](configArguments) new Files(properties) case _ => throw new NoSuchElementException(s"Unknown Source ${keyName.toLowerCase}") } } } import Source val

Spark not working with pureconfig

由 ￡可爱￡侵袭症＋ 提交于 2019-11-28 08:33:37
问题 I'm trying to use pureConfig and configFactory for my spark application configuration. here is my code: import pureconfig.{loadConfigOrThrow} object Source{ def apply(keyName: String, configArguments: Config): Source = { keyName.toLowerCase match { case "mysql" => val properties = loadConfigOrThrow[DBConnectionProperties](configArguments) new MysqlSource(None, properties) case "files" => val properties = loadConfigOrThrow[FilesSourceProperties](configArguments) new Files(properties) case _ =>
订阅 pureconfig