pyspark

问题 I have a dataframe with column as Date along with few other columns. I wanted to validate Date column value and check if the format is of "dd/MM/yyyy". If Date column holds any other format than should mark it as bad record. So I am using option option("dateFormat", "dd/MM/yyyy") to accept date in mentioned format and it accepts the date properly in format "dd/MM/yyyy", but if I pass invalid format (YYYY/mm/dd) still record is not marking as invalid and passed date is converting to garbage

问题 I have a dataframe with column as Date along with few other columns. I wanted to validate Date column value and check if the format is of "dd/MM/yyyy". If Date column holds any other format than should mark it as bad record. So I am using option option("dateFormat", "dd/MM/yyyy") to accept date in mentioned format and it accepts the date properly in format "dd/MM/yyyy", but if I pass invalid format (YYYY/mm/dd) still record is not marking as invalid and passed date is converting to garbage

问题 I have a table in my AWS Glue Data Catalog called 'mytable'. This table is in an on-premises Oracle database connection 'mydb'. I'd like to filter the resulting DynamicFrame to only rows where the X_DATETIME_INSERT column (which is a timestamp) is greater than a certain time (in this case, '2018-05-07 04:00:00'). Afterwards, I'm trying to count the rows to ensure that the count is low (the table is about 40,000 rows, but only a few rows should meet the filter criteria). Here is my current

问题 I have a table in my AWS Glue Data Catalog called 'mytable'. This table is in an on-premises Oracle database connection 'mydb'. I'd like to filter the resulting DynamicFrame to only rows where the X_DATETIME_INSERT column (which is a timestamp) is greater than a certain time (in this case, '2018-05-07 04:00:00'). Afterwards, I'm trying to count the rows to ensure that the count is low (the table is about 40,000 rows, but only a few rows should meet the filter criteria). Here is my current

问题 Spark 2.4 introduced the new SQL function slice , which can be used extract a certain range of elements from an array column. I want to define that range dynamically per row, based on an Integer column that has the number of elements I want to pick from that column. However, simply passing the column to the slice function fails, the function appears to expect integers for start and end values. Is there a way of doing this without writing a UDF? To visualize the problem with an example: I have

问题 Spark 2.4 introduced the new SQL function slice , which can be used extract a certain range of elements from an array column. I want to define that range dynamically per row, based on an Integer column that has the number of elements I want to pick from that column. However, simply passing the column to the slice function fails, the function appears to expect integers for start and end values. Is there a way of doing this without writing a UDF? To visualize the problem with an example: I have

问题 I am saving a dataframe to a CSV file in PySpark using below statement: df_all.repartition(1).write.csv("xyz.csv", header=True, mode='overwrite') But i am getting below error Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/opt/spark-2.3.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 218, in main func, profiler, deserializer, serializer = read_udfs(pickleSer, infile, eval_type) File "/opt/spark-2.3.0-bin-hadoop2.7/python/lib

问题 I am saving a dataframe to a CSV file in PySpark using below statement: df_all.repartition(1).write.csv("xyz.csv", header=True, mode='overwrite') But i am getting below error Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/opt/spark-2.3.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 218, in main func, profiler, deserializer, serializer = read_udfs(pickleSer, infile, eval_type) File "/opt/spark-2.3.0-bin-hadoop2.7/python/lib

问题 I am saving a dataframe to a CSV file in PySpark using below statement: df_all.repartition(1).write.csv("xyz.csv", header=True, mode='overwrite') But i am getting below error Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/opt/spark-2.3.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 218, in main func, profiler, deserializer, serializer = read_udfs(pickleSer, infile, eval_type) File "/opt/spark-2.3.0-bin-hadoop2.7/python/lib

问题 This question already has answers here : Adding a new column in Data Frame derived from other columns (Spark) (3 answers) Closed 2 years ago . I have a data frame in pyspark like sample below. I would like to duplicate a column in the data frame and rename to another column name. Name Age Rate Aira 23 90 Ben 32 98 Cat 27 95 Desired output is : Name Age Rate Rate2 Aira 23 90 90 Ben 32 98 98 Cat 27 95 95 How can I do it? 回答1: Just df.withColumn("Rate2", df["Rate"]) or (in SQL) SELECT *, Rate AS