apache-spark

问题 Apparently dbutils cannot be used in cmd-line spark-submits, you must use Jar Jobs for that, but I MUST use spark-submit style jobs due to other requirements, yet still have a need to list and iterate over file keys in dbfs to make some decisions about which files to use as input to a process... Using scala, what lib in spark or hadoop can I use to retrieve a list of dbfs:/filekeys of a particular pattern? import org.apache.hadoop.fs.Path import org.apache.spark.sql.SparkSession def ls

问题 I want to run my code on a Cluster: my code: import java.util.Properties import edu.stanford.nlp.ling.CoreAnnotations._ import edu.stanford.nlp.pipeline._ import org.apache.spark.{SparkConf, SparkContext} import scala.collection.JavaConversions._ import scala.collection.mutable.ArrayBuffer object Pre2 { def plainTextToLemmas(text: String, pipeline: StanfordCoreNLP): Seq[String] = { val doc = new Annotation(text) pipeline.annotate(doc) val lemmas = new ArrayBuffer[String]() val sentences = doc

问题 I want to run my code on a Cluster: my code: import java.util.Properties import edu.stanford.nlp.ling.CoreAnnotations._ import edu.stanford.nlp.pipeline._ import org.apache.spark.{SparkConf, SparkContext} import scala.collection.JavaConversions._ import scala.collection.mutable.ArrayBuffer object Pre2 { def plainTextToLemmas(text: String, pipeline: StanfordCoreNLP): Seq[String] = { val doc = new Annotation(text) pipeline.annotate(doc) val lemmas = new ArrayBuffer[String]() val sentences = doc

问题 I want to run my code on a Cluster: my code: import java.util.Properties import edu.stanford.nlp.ling.CoreAnnotations._ import edu.stanford.nlp.pipeline._ import org.apache.spark.{SparkConf, SparkContext} import scala.collection.JavaConversions._ import scala.collection.mutable.ArrayBuffer object Pre2 { def plainTextToLemmas(text: String, pipeline: StanfordCoreNLP): Seq[String] = { val doc = new Annotation(text) pipeline.annotate(doc) val lemmas = new ArrayBuffer[String]() val sentences = doc

问题 The databricks docs say that you can change zordering of a delta table by doing: spark.read.table(connRandom) .write.format("delta").saveAsTable(connZorder) sql(s"OPTIMIZE $connZorder ZORDER BY (src_ip, src_port, dst_ip, dst_port)") The problem with this is the switching between the scala and SQL api which is gross. What I want to be able to do is: spark.read.table(connRandom) .write.format("delta").saveAsTable(connZorder) .optimize.zorderBy("src_ip", "src_port", "dst_ip", "dst_port") but I

问题 The databricks docs say that you can change zordering of a delta table by doing: spark.read.table(connRandom) .write.format("delta").saveAsTable(connZorder) sql(s"OPTIMIZE $connZorder ZORDER BY (src_ip, src_port, dst_ip, dst_port)") The problem with this is the switching between the scala and SQL api which is gross. What I want to be able to do is: spark.read.table(connRandom) .write.format("delta").saveAsTable(connZorder) .optimize.zorderBy("src_ip", "src_port", "dst_ip", "dst_port") but I

问题 The databricks docs say that you can change zordering of a delta table by doing: spark.read.table(connRandom) .write.format("delta").saveAsTable(connZorder) sql(s"OPTIMIZE $connZorder ZORDER BY (src_ip, src_port, dst_ip, dst_port)") The problem with this is the switching between the scala and SQL api which is gross. What I want to be able to do is: spark.read.table(connRandom) .write.format("delta").saveAsTable(connZorder) .optimize.zorderBy("src_ip", "src_port", "dst_ip", "dst_port") but I

问题 The databricks docs say that you can change zordering of a delta table by doing: spark.read.table(connRandom) .write.format("delta").saveAsTable(connZorder) sql(s"OPTIMIZE $connZorder ZORDER BY (src_ip, src_port, dst_ip, dst_port)") The problem with this is the switching between the scala and SQL api which is gross. What I want to be able to do is: spark.read.table(connRandom) .write.format("delta").saveAsTable(connZorder) .optimize.zorderBy("src_ip", "src_port", "dst_ip", "dst_port") but I

问题 I'm trying to transform spark dataframe with 10k rows by latest spark 3.0.1 function mapInPandas. Expected output: mapped pandas_function() transforms one row to three, so output transformed_df should have 30k rows Current output: I'm getting 3 rows with 1 core and 24 rows with 8 cores. INPUT: respond_sdf has 10k rows +-----+-------------------------------------------------------------------+ |url |content | +-----+-------------------------------------------------------------------+ |api_1|{

问题 I'm trying to transform spark dataframe with 10k rows by latest spark 3.0.1 function mapInPandas. Expected output: mapped pandas_function() transforms one row to three, so output transformed_df should have 30k rows Current output: I'm getting 3 rows with 1 core and 24 rows with 8 cores. INPUT: respond_sdf has 10k rows +-----+-------------------------------------------------------------------+ |url |content | +-----+-------------------------------------------------------------------+ |api_1|{