spark-dataframe

This question already has an answer here : Prepend zeros to a value in PySpark (1 answer) Closed last year . In short, I'm leveraging spark-xml to do some parsing of XML files. However, using this is removing the leading zeros in all the values I'm interested in. However, I need the final output, which is a DataFrame, to include the leading zeros. I'm unsure/can not figure out a way to add leading zeros to the columns I'm interested in. val df = spark.read .format("com.databricks.spark.xml") .option("rowTag", "output") .option("excludeAttribute", true) .option("allowNumericLeadingZeros", true)

I am getting following error on reading a large 6gb single line json file: Job aborted due to stage failure: Task 5 in stage 0.0 failed 1 times, most recent failure: Lost task 5.0 in stage 0.0 (TID 5, localhost): java.io.IOException: Too many bytes before newline: 2147483648 spark does not read json files with new lines hence the entire 6 gb json file is on a single line: jf = sqlContext.read.json("jlrn2.json") configuration: spark.driver.memory 20g Yep, you have more than Integer.MAX_VALUE bytes in your line. You need to split it up. Keep in mind that Spark is expecting each line to be a

I have a very large pyspark data frame. I need to convert the dataframe into a JSON formatted string for each row then publish the string to a Kafka topic. I originally used the following code. for message in df.toJSON().collect(): kafkaClient.send(message) However the dataframe is very large so it fails when trying to collect() . I was thinking of using a UDF since it processes it row by row. from pyspark.sql.functions import udf, struct def get_row(row): json = row.toJSON() kafkaClient.send(message) return "Sent" send_row_udf = F.udf(get_row, StringType()) df_json = df.withColumn("Sent", get

Using Spark SQL, I have two dataframes, they are created from one, such as: df = sqlContext.createDataFrame(...); df1 = df.filter("value = 'abc'"); //[path, value] df2 = df.filter("value = 'qwe'"); //[path, value] I want to filter df1, if part of its 'path' is any path in df2. So if df1 has row with path 'a/b/c/d/e' I would find out if in df2 is a row that path is 'a/b/c'. In SQL it should be like SELECT * FROM df1 WHERE udf(path) IN (SELECT path FROM df2) where udf is user defined function that shorten original path from df1. Naive solution is to use JOIN and then filter result, but it is

Does anyone know why using Python3's functools.reduce() would lead to worse performance when joining multiple PySpark DataFrames than just iteratively joining the same DataFrames using a for loop? Specifically, this gives a massive slowdown followed by an out-of-memory error: def join_dataframes(list_of_join_columns, left_df, right_df): return left_df.join(right_df, on=list_of_join_columns) joined_df = functools.reduce( functools.partial(join_dataframes, list_of_join_columns), list_of_dataframes, ) whereas this one doesn't: joined_df = list_of_dataframes[0] joined_df.cache() for right_df in