pyspark

问题 I have the following schema for one of columns that I'm processing, |-- time_to_resolution_remainingTime: struct (nullable = true) | |-- _links: struct (nullable = true) | | |-- self: string (nullable = true) | |-- completedCycles: array (nullable = true) | | |-- element: struct (containsNull = true) | | | |-- breached: boolean (nullable = true) | | | |-- elapsedTime: struct (nullable = true) | | | | |-- friendly: string (nullable = true) | | | | |-- millis: long (nullable = true) | | | |--

问题 this is my problem: I have to run a Sagemaker processing job using custom code written in PySpark. I've used the Sagemaker SDK by running these commands: spark_processor = sagemaker.spark.processing.PySparkProcessor( base_job_name="spark-preprocessor", framework_version="2.4", role=role_arn, instance_count=2, instance_type="ml.m5.xlarge", max_runtime_in_seconds=1800, ) spark_processor.run( submit_app="processing.py", arguments=['s3_input_bucket', bucket_name, 's3_input_file_path', file_path ]

问题 i am trying to run a spark script on EMR with custom conda env, 1. created a booststap for conda setup and supplied to the EMR, i don't see any issues with bootstrap but when i do spark-submit it gives me same error no sure what am i missing Traceback (most recent call last): File "/mnt/tmp/spark-b334133c-d22d-42d4-beba-b85fffbbc9c7/iris_cube_analysis.py", line 3, in <module> import iris ImportError: No module named iris spark-submit - spark-submit --deploy-mode client --master yarn --conf

问题 I'm using pyspark df = self.sqlContext.read.option( "es.resource", indexes ).format("org.elasticsearch.spark.sql").load() df = df.filter( df.data.timestamp >= self.period_start ) I'd like to see sql query version of df if possible. something like print(df.query) to see something like select * from my-indexes where data.timestamp > self.period_start 回答1: You can check out this piece of documentation for pyspark.sql.DataFrame.explain . explain prints the (logical and physical) plan to the

问题 My source parquet file has everything as string. My destination parquet file needs to convert this to different datatype like int, string, date etc. How do I do this? 回答1: you may wanted to apply userdefined schema to speedup data loading. There are 2 ways to apply that- using the input DDL-formatted string spark.read.schema("a INT, b STRING, c DOUBLE").parquet("test.parquet") Use StructType schema customSchema = StructType([ StructField("a", IntegerType(), True), StructField("b", StringType(

问题 I'm trying to cast string columns to numeric, but I am getting an exception in PySpark. I provide below the code and the error message. Is it possible to import the specific columns from the csv file as numeric? (the default is to be imported as strings). What are my alternative? My code and the error messages follow below: import pandas as pd import seaborn as sns import findspark findspark.init() import pyspark from pyspark.sql import SparkSession # Loads data. Be careful of indentations

问题 I am trying to read partitioned Avro data which is partitioned based on Year, Month and Day and that seems to be significantly slower than pointing it directly to the path. In the Physical plan I can see that the partition filters are getting passed on, so it is not scanning the entire set of directories but still it is significantly slower. E.g. reading the partitioned data like this profitLossPath="abfss://raw@"+datalakename+".dfs.core.windows.net/datawarehouse/CommercialDM.ProfitLoss/"

问题 I am trying to read partitioned Avro data which is partitioned based on Year, Month and Day and that seems to be significantly slower than pointing it directly to the path. In the Physical plan I can see that the partition filters are getting passed on, so it is not scanning the entire set of directories but still it is significantly slower. E.g. reading the partitioned data like this profitLossPath="abfss://raw@"+datalakename+".dfs.core.windows.net/datawarehouse/CommercialDM.ProfitLoss/"

问题 We are using sagemaker instance to connect to EMR in AWS. We are having some pyspark scripts that unloads athena tables and processes them as part of pipeline. We are using athena tables using glue catalog but when we try to run the job via spark submit, our job fails Code snippet from pyspark import SparkContext, SparkConf from pyspark.context import SparkContext from pyspark.sql import Row, SQLContext, SparkSession import pyspark.sql.dataframe def process_data(): conf = SparkConf()

问题 I saved a pre-trained model from spark-nlp, then I'm trying to run a Python script in Pycharm with anaconda env: Model_path = "./xxx" model = PipelineModel.load(Model_path) But I got the following error: (I tried with pyspark 2.4.4 & spark-nlp2.4.4, and pyspark 2.4.4 & spark-nlp2.5.4) Got the same error: 21/02/05 13:31:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache