pyspark

问题 I'm new to Spark and working with JSON and I'm having trouble doing something fairly simple (I think). I've tried using parts of solutions to similar questions but can't quite get it right. I currently have a Spark dataframe with several columns representing variables. Each row is a unique combination of variable values. I then have a UDF that is applied to every row which takes each of the columns as input, does some analysis, and outputs a summary table as a JSON string for each row, and

问题 I'm new to Spark and working with JSON and I'm having trouble doing something fairly simple (I think). I've tried using parts of solutions to similar questions but can't quite get it right. I currently have a Spark dataframe with several columns representing variables. Each row is a unique combination of variable values. I then have a UDF that is applied to every row which takes each of the columns as input, does some analysis, and outputs a summary table as a JSON string for each row, and

问题 I'm looking for a way to aggregate by hour my data. I want firstly to keep only hours in my evtTime. My DataFrame looks like this: Row(access=u'WRITE', agentHost=u'xxxxxx50.haas.xxxxxx', cliIP=u'192.000.00.000', enforcer=u'ranger-acl', event_count=1, event_dur_ms=0, evtTime=u'2017-10-01 23:03:51.337', id=u'a43d824c-1e53-439b-b374-96b76bacf714', logType=u'RangerAudit', policy=699, reason=u'/project-h/xxxx/xxxx/warehouse/rocq.db/f_crcm_res_temps_retrait', repoType=1, reqUser=u'rocqphadm',

问题 I'm looking for a way to aggregate by hour my data. I want firstly to keep only hours in my evtTime. My DataFrame looks like this: Row(access=u'WRITE', agentHost=u'xxxxxx50.haas.xxxxxx', cliIP=u'192.000.00.000', enforcer=u'ranger-acl', event_count=1, event_dur_ms=0, evtTime=u'2017-10-01 23:03:51.337', id=u'a43d824c-1e53-439b-b374-96b76bacf714', logType=u'RangerAudit', policy=699, reason=u'/project-h/xxxx/xxxx/warehouse/rocq.db/f_crcm_res_temps_retrait', repoType=1, reqUser=u'rocqphadm',

问题 Pyspark n00b... How do I replace a column with a substring of itself? I'm trying to remove a select number of characters from the start and end of string. from pyspark.sql.functions import substring import pandas as pd pdf = pd.DataFrame({'COLUMN_NAME':['_string_','_another string_']}) # this is what i'm looking for... pdf['COLUMN_NAME_fix']=pdf['COLUMN_NAME'].str[1:-1] df = sqlContext.createDataFrame(pdf) # following not working... COLUMN_NAME_fix is blank df.withColumn('COLUMN_NAME_fix',

问题 Pyspark n00b... How do I replace a column with a substring of itself? I'm trying to remove a select number of characters from the start and end of string. from pyspark.sql.functions import substring import pandas as pd pdf = pd.DataFrame({'COLUMN_NAME':['_string_','_another string_']}) # this is what i'm looking for... pdf['COLUMN_NAME_fix']=pdf['COLUMN_NAME'].str[1:-1] df = sqlContext.createDataFrame(pdf) # following not working... COLUMN_NAME_fix is blank df.withColumn('COLUMN_NAME_fix',

问题 I have a spark data frame which I want to divide into train, validation and test in the ratio 0.60, 0.20,0.20. I used the following code for the same: def data_split(x): global data_map_var d_map = data_map_var.value data_row = x.asDict() import random rand = random.uniform(0.0,1.0) ret_list = () if rand <= 0.6: ret_list = (data_row['TRANS'] , d_map[data_row['ITEM']] , data_row['Ratings'] , 'train') elif rand <=0.8: ret_list = (data_row['TRANS'] , d_map[data_row['ITEM']] , data_row['Ratings']

问题 I have a spark data frame which I want to divide into train, validation and test in the ratio 0.60, 0.20,0.20. I used the following code for the same: def data_split(x): global data_map_var d_map = data_map_var.value data_row = x.asDict() import random rand = random.uniform(0.0,1.0) ret_list = () if rand <= 0.6: ret_list = (data_row['TRANS'] , d_map[data_row['ITEM']] , data_row['Ratings'] , 'train') elif rand <=0.8: ret_list = (data_row['TRANS'] , d_map[data_row['ITEM']] , data_row['Ratings']

问题 This question already has answers here : Pyspark dataframe operator “IS NOT IN” (6 answers) Closed last year . I am trying to get all rows within a dataframe where a columns value is not within a list (so filtering by exclusion). As an example: df = sqlContext.createDataFrame([('1','a'),('2','b'),('3','b'),('4','c'),('5','d')] ,schema=('id','bar')) I get the data frame: +---+---+ | id|bar| +---+---+ | 1| a| | 2| b| | 3| b| | 4| c| | 5| d| +---+---+ I only want to exclude rows where bar is ('a

问题 After joining two dataframes (which have their own ID's) I have some duplicates (repeated ID's from both sources) I want to drop all rows that are duplicates on either ID (so not retain a single occurrence of a duplicate) I can group by the first ID, do a count and filter for count ==1, then repeat that for the second ID, then inner join these outputs back to the original joined dataframe - but this feels a bit long. Is there a simpler method like dropDuplicates() but where none of the