pyspark-sql

I'm trying to do some NLP text clean up of some Unicode columns in a PySpark DataFrame. I've tried in Spark 1.3, 1.5 and 1.6 and can't seem to get things to work for the life of me. I've also tried using Python 2.7 and Python 3.4. I've created an extremely simple udf as seen below that should just return a string back for each record in a new column. Other functions will manipulate the text and then return the changed text back in a new column. import pyspark from pyspark.sql import SQLContext from pyspark.sql.types import * from pyspark.sql import SQLContext from pyspark.sql.functions import

I am new to spark and was trying out a few commands in sparkSql using python when I came across these two commands: createOrReplaceTempView () and registerTempTable (). What is the difference between the two commands?. They seem to have same set of functionalities. registerTempTable is a part of the 1.x API and has been deprecated in Spark 2.0. createOrReplaceTempView and createTempView have been introduced in Spark 2.0, as a replacement for registerTempTable . Other than that registerTempTable and createOrReplaceTempView functionally equivalent and the former one calls the latter one. No

I need to pivot more than one column in a pyspark dataframe. Sample dataframe, >>> d = [(100,1,23,10),(100,2,45,11),(100,3,67,12),(100,4,78,13),(101,1,23,10),(101,2,45,13),(101,3,67,14),(101,4,78,15),(102,1,23,10),(102,2,45,11),(102,3,67,16),(102,4,78,18)] >>> mydf = spark.createDataFrame(d,['id','day','price','units']) >>> mydf.show() +---+---+-----+-----+ | id|day|price|units| +---+---+-----+-----+ |100| 1| 23| 10| |100| 2| 45| 11| |100| 3| 67| 12| |100| 4| 78| 13| |101| 1| 23| 10| |101| 2| 45| 13| |101| 3| 67| 14| |101| 4| 78| 15| |102| 1| 23| 10| |102| 2| 45| 11| |102| 3| 67| 16| |102| 4|

I'm trying to migrate from Spark 1.6.1 to Spark 2.0.0 and I am getting a weird error when trying to read a csv file into SparkSQL. Previously, when I would read a file from local disk in pyspark I would do: Spark 1.6 df = sqlContext.read \ .format('com.databricks.spark.csv') \ .option('header', 'true') \ .load('file:///C:/path/to/my/file.csv', schema=mySchema) In the latest release I think it should look like this: Spark 2.0 spark = SparkSession.builder \ .master('local[*]') \ .appName('My App') \ .getOrCreate() df = spark.read \ .format('csv') \ .option('header', 'true') \ .load('file:///C:

This question already has an answer here: Create Spark DataFrame. Can not infer schema for type: <type 'float'> 1 answer The case is really simple, I need to convert a python list into data frame with following code from pyspark.sql.types import StructType from pyspark.sql.types import StructField from pyspark.sql.types import StringType, IntegerType schema = StructType([StructField("value", IntegerType(), True)]) my_list = [1, 2, 3, 4] rdd = sc.parallelize(my_list) df = sqlContext.createDataFrame(rdd, schema) df.show() it failed with following error: raise TypeError("StructType can not accept

when i try to feed df2 to kmeans i get the following error clusters = KMeans.train(df2, 10, maxIterations=30, runs=10, initializationMode="random") The error i get: Cannot convert type <class 'pyspark.sql.types.Row'> into Vector df2 is a dataframe created as follow: df = sqlContext.read.json("data/ALS3.json") df2 = df.select('latitude','longitude') df2.show() latitude| longitude| 60.1643075| 24.9460844| 60.4686748| 22.2774728| how can i convert this two columns to Vector and feed it to KMeans? ML The problem is that you missed the documentation's example , and it's pretty clear that the method