pyspark

问题 Even if it is a Hive table or an HDFS file, when Spark reads the data and creates a dataframe, I was thinking that the number of partitions in the RDD/dataframe will be equal to the number of partfiles in HDFS. But when I did a test with Hive external table, I could see that the number was coming different than the number of part-files .The number of partitions in a dataframe was 119. The table was a Hive partitioned table with 150 partfiles in it, with a minimum size of a file 30 MB and max

问题 Even if it is a Hive table or an HDFS file, when Spark reads the data and creates a dataframe, I was thinking that the number of partitions in the RDD/dataframe will be equal to the number of partfiles in HDFS. But when I did a test with Hive external table, I could see that the number was coming different than the number of part-files .The number of partitions in a dataframe was 119. The table was a Hive partitioned table with 150 partfiles in it, with a minimum size of a file 30 MB and max

问题 I posted this question earlier and got some advice to use PySpark instead. How can I merge this large dataset into one large dataframe efficiently? The following zip file (https://fred.stlouisfed.org/categories/32263/downloaddata/INTRNTL_csv_2.zip) contains a folder called data with around 130,000 of csv files. I want to merge all of them into one single dataframe. I have 16gb of RAM and I keep running out of RAM when I hit the first few hundred files. The files' total size is only about 300

问题 I have a Pyspark dataframe with some non-unique key key and some columns number and value . For most keys , the number column goes from 1 to 12, but for some of them, there are gaps in numbers (for ex. we have numbers [1, 2, 5, 9] ). I would like to add missing rows, so that for every key we have all the numbers in range 1-12 populated with the last seen value. So that for table key number value a 1 6 a 2 10 a 5 20 a 9 25 I would like to get key number value a 1 6 a 2 10 a 3 10 a 4 10 a 5 20

问题 I am trying to change all the columns of a spark dataframe to double type but i want to know if there is a better way of doing it than just looping over the columns and casting. 回答1: With this dataframe: df = spark.createDataFrame( [ (1,2), (2,3), ], ["foo","bar"] ) df.show() +---+---+ |foo|bar| +---+---+ | 1| 2| | 2| 3| +---+---+ the for loop is problably the easiest and more natural solution. from pyspark.sql import functions as F for col in df.columns: df = df.withColumn( col, F.col(col)

问题 I am trying to change all the columns of a spark dataframe to double type but i want to know if there is a better way of doing it than just looping over the columns and casting. 回答1: With this dataframe: df = spark.createDataFrame( [ (1,2), (2,3), ], ["foo","bar"] ) df.show() +---+---+ |foo|bar| +---+---+ | 1| 2| | 2| 3| +---+---+ the for loop is problably the easiest and more natural solution. from pyspark.sql import functions as F for col in df.columns: df = df.withColumn( col, F.col(col)

问题 I'm running a pyspark 2.2.0 job using the Apache Spark local mode and see the following warning: WARN RowBasedKeyValueBatch: Calling spill() on RowBasedKeyValueBatch. Will not spill but return 0. What could be the reason for this warning? Is this something I should care about or can I safely ignore it? 回答1: As indicated here this warning means that your RAM is full and that part of the RAM contents are moved to disk. See also the Spark FAQ Does my data need to fit in memory to use Spark? No.

问题 This question already has answers here : Spark How to Specify Number of Resulting Files for DataFrame While/After Writing (1 answer) How to control the number of output part files created by Spark job upon writing? (2 answers) Closed 17 days ago . When i save the DataFrame resulting from sparksql query in HDFS, it generates large number of part files with each one at 1.4 KB. is there a way to increase size of file as every part file contains about 2 records. df_crimes_dates_formated = spark

问题 I have a Docker container that's running Apache Spark with a master and a slave worker. I'm attempting to submit a job from a Jupyter notebook on the host machine. See below: # Init !pip install findspark import findspark findspark.init() # Context setup from pyspark import SparkConf, SparkContext # Docker container is exposing port 7077 conf = SparkConf().setAppName('test').setMaster('spark://localhost:7077') sc = SparkContext(conf=conf) sc # Execute step import random num_samples = 1000 def

问题 I have a Docker container that's running Apache Spark with a master and a slave worker. I'm attempting to submit a job from a Jupyter notebook on the host machine. See below: # Init !pip install findspark import findspark findspark.init() # Context setup from pyspark import SparkConf, SparkContext # Docker container is exposing port 7077 conf = SparkConf().setAppName('test').setMaster('spark://localhost:7077') sc = SparkContext(conf=conf) sc # Execute step import random num_samples = 1000 def