pyspark-sql

问题 I am able to execute a simple SQL statement using PySpark in Azure Databricks but I want to execute a stored procedure instead. Below is the PySpark code I tried. #initialize pyspark import findspark findspark.init('C:\Spark\spark-2.4.5-bin-hadoop2.7') #import required modules from pyspark import SparkConf, SparkContext from pyspark.sql import SparkSession from pyspark.sql import * import pandas as pd #Create spark configuration object conf = SparkConf() conf.setMaster("local").setAppName("My

问题 I am able to execute a simple SQL statement using PySpark in Azure Databricks but I want to execute a stored procedure instead. Below is the PySpark code I tried. #initialize pyspark import findspark findspark.init('C:\Spark\spark-2.4.5-bin-hadoop2.7') #import required modules from pyspark import SparkConf, SparkContext from pyspark.sql import SparkSession from pyspark.sql import * import pandas as pd #Create spark configuration object conf = SparkConf() conf.setMaster("local").setAppName("My

问题 I am able to execute a simple SQL statement using PySpark in Azure Databricks but I want to execute a stored procedure instead. Below is the PySpark code I tried. #initialize pyspark import findspark findspark.init('C:\Spark\spark-2.4.5-bin-hadoop2.7') #import required modules from pyspark import SparkConf, SparkContext from pyspark.sql import SparkSession from pyspark.sql import * import pandas as pd #Create spark configuration object conf = SparkConf() conf.setMaster("local").setAppName("My

问题 I am able to execute a simple SQL statement using PySpark in Azure Databricks but I want to execute a stored procedure instead. Below is the PySpark code I tried. #initialize pyspark import findspark findspark.init('C:\Spark\spark-2.4.5-bin-hadoop2.7') #import required modules from pyspark import SparkConf, SparkContext from pyspark.sql import SparkSession from pyspark.sql import * import pandas as pd #Create spark configuration object conf = SparkConf() conf.setMaster("local").setAppName("My

问题 I am trying to add a column to DataFrame depending on whether column value is in another column as follow: df=df.withColumn('new_column',when(df['color']=='blue'|df['color']=='green','A').otherwise('WD')) after running the code I obtain the following error: Py4JError: An error occurred while calling o59.or. Trace: py4j.Py4JException: Method or([class java.lang.String]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine

问题 My input dataframe looks like the below from pyspark.sql import SparkSession spark = SparkSession.builder.appName("Basics").getOrCreate() df=spark.createDataFrame(data=[('Alice',4.300,None),('Bob',float('nan'),897)],schema=['name','High','Low']) +-----+----+----+ | name|High| Low| +-----+----+----+ |Alice| 4.3|null| | Bob| NaN| 897| +-----+----+----+ Expected Output if divided by 10.0 +-----+----+----+ | name|High| Low| +-----+----+----+ |Alice| 0.43|null| | Bob| NaN| 89.7| +-----+----+----+

问题 My input dataframe looks like the below from pyspark.sql import SparkSession spark = SparkSession.builder.appName("Basics").getOrCreate() df=spark.createDataFrame(data=[('Alice',4.300,None),('Bob',float('nan'),897)],schema=['name','High','Low']) +-----+----+----+ | name|High| Low| +-----+----+----+ |Alice| 4.3|null| | Bob| NaN| 897| +-----+----+----+ Expected Output if divided by 10.0 +-----+----+----+ | name|High| Low| +-----+----+----+ |Alice| 0.43|null| | Bob| NaN| 89.7| +-----+----+----+

问题 My input dataframe looks like the below from pyspark.sql import SparkSession spark = SparkSession.builder.appName("Basics").getOrCreate() df=spark.createDataFrame(data=[('Alice',4.300,None),('Bob',float('nan'),897)],schema=['name','High','Low']) +-----+----+----+ | name|High| Low| +-----+----+----+ |Alice| 4.3|null| | Bob| NaN| 897| +-----+----+----+ Expected Output if divided by 10.0 +-----+----+----+ | name|High| Low| +-----+----+----+ |Alice| 0.43|null| | Bob| NaN| 89.7| +-----+----+----+

问题 I have fixed width file as below 00120181120xyz12341 00220180203abc56792 00320181203pqr25483 And a corresponding JSON file that specifies the schema: {"Column":"id","From":"1","To":"3"} {"Column":"date","From":"4","To":"8"} {"Column":"name","From":"12","To":"3"} {"Column":"salary","From":"15","To":"5"} I read the schema file into DataFrame using: SchemaFile = spark.read\ .format("json")\ .option("header","true")\ .json('C:\Temp\schemaFile\schema.json') SchemaFile.show() #+------+----+---+ #

问题 I'm trying to cast string columns to numeric, but I am getting an exception in PySpark. I provide below the code and the error message. Is it possible to import the specific columns from the csv file as numeric? (the default is to be imported as strings). What are my alternative? My code and the error messages follow below: import pandas as pd import seaborn as sns import findspark findspark.init() import pyspark from pyspark.sql import SparkSession # Loads data. Be careful of indentations