apache-spark

问题 I can't find an easy and elegant solution to this one. I have a df1 with this column : |-- guitars: array (nullable = true) | |-- element: long (containsNull = true) I have a df2 made of guitars, and an id matching with the Long in my df 1. root |-- guitarId: long (nullable = true) |-- make: string (nullable = true) |-- model: string (nullable = true) |-- type: string (nullable = true) I want to join my two dfs, obviously, and instead of having an array of long, I want an array of struct

问题 I can't find an easy and elegant solution to this one. I have a df1 with this column : |-- guitars: array (nullable = true) | |-- element: long (containsNull = true) I have a df2 made of guitars, and an id matching with the Long in my df 1. root |-- guitarId: long (nullable = true) |-- make: string (nullable = true) |-- model: string (nullable = true) |-- type: string (nullable = true) I want to join my two dfs, obviously, and instead of having an array of long, I want an array of struct

问题 I have a Spark Dataframe where each row has a review. +--------------------+ | reviewText| +--------------------+ |Spiritually and m...| |This is one my mu...| |This book provide...| |I first read THE ...| +--------------------+ I have tried: SplitSentences = df.withColumn("split_sent",sentencesplit_udf(col('reviewText'))) SplitSentences = SplitSentences.select(SplitSentences.split_sent) Then I created the function: def word_count(text): return len(text.split()) wordcount_udf = udf(lambda x:

问题 Suppose I have this DataFrame ( df ): user food affinity 'u1' 'pizza' 5 'u1' 'broccoli' 3 'u1' 'ice cream' 4 'u2' 'pizza' 1 'u2' 'broccoli' 3 'u2' 'ice cream' 1 Namely each user has a certain (computed) affinity to a series of foods. The DataFrame is built from several What I need to do is create a JSON file for each user , with their affinities. For instance, for user 'u1', I want to have file for user 'u1' containing [ {'food': 'pizza', 'affinity': 5}, {'food': 'broccoli', 'affinity': 3}, {

问题 I'm trying to work with JSON file on spark (pyspark) environment. Problem: Unable to convert JSON to expected format in Pyspark Dataframe 1st Input data set: https://health.data.ny.gov/api/views/cnih-y5dw/rows.json In this file metadata is defined at start for of the file with tag "meta" and then followed by data with tag "data". FYI: Steps taken to download data from web to local drive. 1. I've downloaded file to my local drive 2. then pushed to hdfs - from there I'm reading it to spark

问题 I'm trying to work with JSON file on spark (pyspark) environment. Problem: Unable to convert JSON to expected format in Pyspark Dataframe 1st Input data set: https://health.data.ny.gov/api/views/cnih-y5dw/rows.json In this file metadata is defined at start for of the file with tag "meta" and then followed by data with tag "data". FYI: Steps taken to download data from web to local drive. 1. I've downloaded file to my local drive 2. then pushed to hdfs - from there I'm reading it to spark

问题 I'm trying to work with JSON file on spark (pyspark) environment. Problem: Unable to convert JSON to expected format in Pyspark Dataframe 1st Input data set: https://health.data.ny.gov/api/views/cnih-y5dw/rows.json In this file metadata is defined at start for of the file with tag "meta" and then followed by data with tag "data". FYI: Steps taken to download data from web to local drive. 1. I've downloaded file to my local drive 2. then pushed to hdfs - from there I'm reading it to spark

问题 I am having difficulties creating a basic spark streaming application. Right now, am trying it on my local machine. I have done following setup. -Setup Zookeeper -Setup Kafka ( Version : kafka_2.10-0.9.0.1) -Created a topic using below command kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test -Started producer and consumer on two different cmd terminals using below commands Producer : kafka-console-producer.bat --broker-list localhost:9092

问题 I'm trying to check which entries in a spark dataframe (column with lists) contain the largest quantity of values from a given list. The best approach I've came up with is iterating over a dataframe with rdd.foreach() and comparing a given list to every entry using python's set1.intersection(set2) . My question is does spark have any built-in functionality for this so iterating with .foreach could be avoided? Thanks for any help! P.S. my dataframe looks like this: +-------------+-------------

问题 I am new to scala..I want to know when processing large datasets with scala in spark is it possible to read as int RDD instead of String RDD I tried the below: val intArr = sc .textFile("Downloads/data/train.csv") .map(line=>line.split(",")) .map(_.toInt) But I am getting the error: error: value toInt is not a member of Array[String] I need to convert to int rdd because down the line i need to do the below val vectors = intArr.map(p => Vectors.dense(p)) which requires the type to be integer