pyspark

问题 I'm using pySpark in version 2.3 (cannot update to 2.4 in my current dev-System) and have the following questions concerning the foreachPartition. First a little context: As far as I understood pySpark- UDFs force the Python-code to be executed outside the Java Virtual Machine (JVM) in a Python-instance, making it performance-costing. Since I need to apply some Python-functions to my data and want to minimize overhead costs, I had the idea to at least load a handable bunch of data into the

问题 I'm using pySpark in version 2.3 (cannot update to 2.4 in my current dev-System) and have the following questions concerning the foreachPartition. First a little context: As far as I understood pySpark- UDFs force the Python-code to be executed outside the Java Virtual Machine (JVM) in a Python-instance, making it performance-costing. Since I need to apply some Python-functions to my data and want to minimize overhead costs, I had the idea to at least load a handable bunch of data into the

问题 Context A dataframe should have the category column, which is based on a set of fixed rules. The set of rules becomes quite large. Question Is there a way to use a list of tuples (see example below) to dynamically chain the when conditions to achieve the same result as hard coded solution at the bottom. # Potential list of rule definitions category_rules = [ ('A', 8, 'small'), ('A', 30, 'large'), ('B', 5, 'small'), # Group, size smaller value --> Category # and so on ... e.g., ] Example Here

问题 I'm trying to import data with parquet format with custom schema but it returns : TypeError: option() missing 1 required positional argument: 'value' ProductCustomSchema = StructType([ StructField("id_sku", IntegerType(), True), StructField("flag_piece", StringType(), True), StructField("flag_weight", StringType(), True), StructField("ds_sku", StringType(), True), StructField("qty_pack", FloatType(), True)]) def read_parquet_(path, schema) : return spark.read.format("parquet")\ .option(schema

问题 I have a spark dataframe that contains sales prediction data for some products in some stores over a time period. How do I calculate the rolling sum of Predictions for a window size of next N values? Input Data +-----------+---------+------------+------------+---+ | ProductId | StoreId | Date | Prediction | N | +-----------+---------+------------+------------+---+ | 1 | 100 | 2019-07-01 | 0.92 | 2 | | 1 | 100 | 2019-07-02 | 0.62 | 2 | | 1 | 100 | 2019-07-03 | 0.89 | 2 | | 1 | 100 | 2019-07-04

问题 I have a spark dataframe that contains sales prediction data for some products in some stores over a time period. How do I calculate the rolling sum of Predictions for a window size of next N values? Input Data +-----------+---------+------------+------------+---+ | ProductId | StoreId | Date | Prediction | N | +-----------+---------+------------+------------+---+ | 1 | 100 | 2019-07-01 | 0.92 | 2 | | 1 | 100 | 2019-07-02 | 0.62 | 2 | | 1 | 100 | 2019-07-03 | 0.89 | 2 | | 1 | 100 | 2019-07-04

问题 I have a spark dataframe that contains sales prediction data for some products in some stores over a time period. How do I calculate the rolling sum of Predictions for a window size of next N values? Input Data +-----------+---------+------------+------------+---+ | ProductId | StoreId | Date | Prediction | N | +-----------+---------+------------+------------+---+ | 1 | 100 | 2019-07-01 | 0.92 | 2 | | 1 | 100 | 2019-07-02 | 0.62 | 2 | | 1 | 100 | 2019-07-03 | 0.89 | 2 | | 1 | 100 | 2019-07-04

问题 Elasticsaerch's documentation only covers loading a complete index to Spark. from pyspark.sql import SQLContext sqlContext = SQLContext(sc) df = sqlContext.read.format("org.elasticsearch.spark.sql").load("index/type") df.printSchema() How can you perform a query to return data from an Elasticsearch index and load them to Spark as a DataFrame using pyspark? 回答1: Below is how I do it. General environment settings and command: export SPARK_HOME=/home/ezerkar/spark-1.6.0-bin-hadoop2.6 export

问题 Elasticsaerch's documentation only covers loading a complete index to Spark. from pyspark.sql import SQLContext sqlContext = SQLContext(sc) df = sqlContext.read.format("org.elasticsearch.spark.sql").load("index/type") df.printSchema() How can you perform a query to return data from an Elasticsearch index and load them to Spark as a DataFrame using pyspark? 回答1: Below is how I do it. General environment settings and command: export SPARK_HOME=/home/ezerkar/spark-1.6.0-bin-hadoop2.6 export

问题 Elasticsaerch's documentation only covers loading a complete index to Spark. from pyspark.sql import SQLContext sqlContext = SQLContext(sc) df = sqlContext.read.format("org.elasticsearch.spark.sql").load("index/type") df.printSchema() How can you perform a query to return data from an Elasticsearch index and load them to Spark as a DataFrame using pyspark? 回答1: Below is how I do it. General environment settings and command: export SPARK_HOME=/home/ezerkar/spark-1.6.0-bin-hadoop2.6 export