pyspark

问题 I want to use dbutils command for accessing secrets in my pyspark job submitting through Spark-Submit inside Jobs on Databricks. While using dbutils command it is giving error dbutils not defined. Is there is the way to use dbutils in a pyspark job other than a notebook? Tried the following solutions: 1) import DBUtils, according to this solution. But this is not Databricks dbutils. 2) import pyspark.dbutils import DBUtils , according to this solution. But this also didn't work. pyspark job

问题 I want to use dbutils command for accessing secrets in my pyspark job submitting through Spark-Submit inside Jobs on Databricks. While using dbutils command it is giving error dbutils not defined. Is there is the way to use dbutils in a pyspark job other than a notebook? Tried the following solutions: 1) import DBUtils, according to this solution. But this is not Databricks dbutils. 2) import pyspark.dbutils import DBUtils , according to this solution. But this also didn't work. pyspark job

问题 I need to reduce a datafame and export it to a parquet. I need to make sure that I have ex. 10000 rows for each value in a column. The dataframe I am working with looks like the following: +-------------+-------------------+ | Make| Model| +-------------+-------------------+ | PONTIAC| GRAND AM| | BUICK| CENTURY| | LEXUS| IS 300| |MERCEDES-BENZ| SL-CLASS| | PONTIAC| GRAND AM| | TOYOTA| PRIUS| | MITSUBISHI| MONTERO SPORT| |MERCEDES-BENZ| SLK-CLASS| | TOYOTA| CAMRY| | JEEP| WRANGLER| |

问题 I am new to pyspark. I am trying to understand how to access parquet file with multiple level of nested struct and array's. I need to replace some value in a data-frame (with nested schema) with null, I have seen this solution it works fine with structs but it not sure how this works with arrays. My schema is something like this |-- unitOfMeasure: struct | |-- raw: struct | | |-- id: string | | |-- codingSystemId: string | | |-- display: string | |-- standard: struct | | |-- id: string | | |-

问题 I have setup Spark2.1.1 cluster (1 master 2 slaves) following http://paxcel.net/blog/how-to-setup-apache-spark-standalone-cluster-on-multiple-machine/ in standalone mode. I do not have a pre-Hadoop setup on of the machine. I wanted to start spark-history server. I run it as follows: roshan@bolt:~/spark/spark_home/sbin$ ./start-history-server.sh and in the spark-defaults.conf I set this: spark.eventLog.enabled true But it fails with the error: 7/06/29 22:59:03 INFO SecurityManager:

问题 Having this dataframe I am getting Column is not iterable when I try to groupBy and getting max: linesWithSparkDF +---+-----+ | id|cycle| +---+-----+ | 31| 26| | 31| 28| | 31| 29| | 31| 97| | 31| 98| | 31| 100| | 31| 101| | 31| 111| | 31| 112| | 31| 113| +---+-----+ only showing top 10 rows ipython-input-41-373452512490> in runlgmodel2(model, data) 65 linesWithSparkDF.show(10) 66 ---> 67 linesWithSparkGDF = linesWithSparkDF.groupBy(col("id")).agg(max(col("cycle"))) 68 print "linesWithSparkGDF

问题 Say I have two large RDD's, A and B, containing key-value pairs. I want to join A and B using the key, but of the pairs (a,b) that match, I only want a tiny fraction of "good" ones. So I do the join and apply a filter afterwards: A.join(B).filter(isGoodPair) where isGoodPair is a boolean function that tells me if a pair (a,b) is good or not. For this to scale well, Spark's scheduler would ideally avoid forming all pairs in A.join(B) explicitly. Even on a massively distributed basis, this

问题 The pyspark RDD documentation http://spark.apache.org/docs/1.2.1/api/python/pyspark.html#pyspark.RDD does not show any method(s) to display partition information for an RDD. Is there any way to get that information without executing an additional step e.g.: myrdd.mapPartitions(lambda x: iter[1]).sum() The above does work .. but seems like extra effort. 回答1: I missed it: very simple: rdd.getNumPartitions() Not used to the java-ish get FooMethod() anymore ;) Update : Adding in the comment from

问题 I want to create a sample DataFrame but the following code is not working: df = spark.createDataFrame(["10","11","13"], ("age")) ## ValueError ## ... ## ValueError: Could not parse datatype: age Expected result is: age 10 11 13 回答1: the following code is not working With single element you need a schema as type spark.createDataFrame(["10","11","13"], "string").toDF("age") or DataType : from pyspark.sql.types import StringType spark.createDataFrame(["10","11","13"], StringType()).toDF("age")

问题 Hi I have the following issue: numeric.registerTempTable("numeric"). All the values that I want to filter on are literal null strings and not N/A or Null values. I tried these three options: numeric_filtered = numeric.filter(numeric['LOW'] != 'null').filter(numeric['HIGH'] != 'null').filter(numeric['NORMAL'] != 'null') numeric_filtered = numeric.filter(numeric['LOW'] != 'null' AND numeric['HIGH'] != 'null' AND numeric['NORMAL'] != 'null') sqlContext.sql("SELECT * from numeric WHERE LOW !=