pyspark

问题 i'm making a loop in pyspark, and i have this message: "Column is not iterable" This is the code: (regexp_replace(data_join_result[varibale_choisie], (random.choice(data_join_result.collect()[j][varibale_choisie])), data_join_result.collect()[j][lettre_choisie] )))) in the error message, the problem comes at this moment: data_join_result.collect()[j][lettre_choisie] My input: VARIABLEA  | VARIABLEB BLUE        | WHITE PINK         | DARK My expected output: VARIABLEA  | VARIABLEB BLTE       

问题 i'm making a loop in pyspark, and i have this message: "Column is not iterable" This is the code: (regexp_replace(data_join_result[varibale_choisie], (random.choice(data_join_result.collect()[j][varibale_choisie])), data_join_result.collect()[j][lettre_choisie] )))) in the error message, the problem comes at this moment: data_join_result.collect()[j][lettre_choisie] My input: VARIABLEA  | VARIABLEB BLUE        | WHITE PINK         | DARK My expected output: VARIABLEA  | VARIABLEB BLTE       

问题 i'm making a loop in pyspark, and i have this message: "Column is not iterable" This is the code: (regexp_replace(data_join_result[varibale_choisie], (random.choice(data_join_result.collect()[j][varibale_choisie])), data_join_result.collect()[j][lettre_choisie] )))) in the error message, the problem comes at this moment: data_join_result.collect()[j][lettre_choisie] My input: VARIABLEA  | VARIABLEB BLUE        | WHITE PINK         | DARK My expected output: VARIABLEA  | VARIABLEB BLTE       

问题 I am using Python 2.6.6 and Spark 1.6.0. I have df like this: id | name | number | -------------------------- 1 | joe | 148590 | 2 | bob | 148590 | 2 | steve | 279109 | 3 | sue | 382901 | 3 | linda | 148590 | Whenever I try to run something like df2 = df.groupBy('id','length','type').pivot('id').agg(F.collect_list('name')) , I get the following error pyspark.sql.utils.AnalysisException: u"Aggregate expression required for pivot, found 'pythonUDF#93';" Why is this? 回答1: Resolved. I used

问题 I am using Python 2.6.6 and Spark 1.6.0. I have df like this: id | name | number | -------------------------- 1 | joe | 148590 | 2 | bob | 148590 | 2 | steve | 279109 | 3 | sue | 382901 | 3 | linda | 148590 | Whenever I try to run something like df2 = df.groupBy('id','length','type').pivot('id').agg(F.collect_list('name')) , I get the following error pyspark.sql.utils.AnalysisException: u"Aggregate expression required for pivot, found 'pythonUDF#93';" Why is this? 回答1: Resolved. I used

问题 I have a pyspark data frame df which is holding large no of rows.Once of the columns is lat-long. I want to find the state name from the lat-long.I am using the below code import reverse_geocoder as rg new_df = df_new2.toPandas() list_long_lat = a["lat_long"].tolist() result = rg.search(list_long_lat) state_name=[] for each_entry in result: state_name.append(each_entry["admin2"]) state_values = pd.Series(state_name) a.insert(loc=0, column='State_name', value=state_values) first of all when

问题 I have a pyspark data frame df which is holding large no of rows.Once of the columns is lat-long. I want to find the state name from the lat-long.I am using the below code import reverse_geocoder as rg new_df = df_new2.toPandas() list_long_lat = a["lat_long"].tolist() result = rg.search(list_long_lat) state_name=[] for each_entry in result: state_name.append(each_entry["admin2"]) state_values = pd.Series(state_name) a.insert(loc=0, column='State_name', value=state_values) first of all when

问题 I have a pyspark data frame df which is holding large no of rows.Once of the columns is lat-long. I want to find the state name from the lat-long.I am using the below code import reverse_geocoder as rg new_df = df_new2.toPandas() list_long_lat = a["lat_long"].tolist() result = rg.search(list_long_lat) state_name=[] for each_entry in result: state_name.append(each_entry["admin2"]) state_values = pd.Series(state_name) a.insert(loc=0, column='State_name', value=state_values) first of all when

问题 I am executing spark-submit job on Hadoop-Yarn cluster. spark-submit /opt/spark/examples/src/main/python/pi.py 1000 but facing below error message. It seems to be worker is not starting. 2018-12-20 07:25:14 INFO SparkContext:54 - Created broadcast 0 from broadcast at DAGScheduler.scala:1161 2018-12-20 07:25:14 INFO DAGScheduler:54 - Submitting 1000 missing tasks from ResultStage 0 (PythonRDD[1] at reduce at /opt/spark/examples/src/main/python/pi.py:44) (first 15 tasks are for partitions

问题 I looked at spark-rdd to dataframe. I read my gziped json into rdd rdd1 =sc.textFile('s3://cw-milenko-tests/Json_gzips/ticr_calculated_2_2020-05-27T11-59-06.json.gz') I want to convert it to spark dataframe. The first method from the linked SO question does not work. This is the first row form the file {"code_event": "1092406", "code_event_system": "LOTTO", "company_id": "2", "date_event": "2020-05-27 12:00:00.000", "date_event_real": "0001-01-01 00:00:00.000", "ecode_class": "", "ecode_event