rdd

问题 What does the meaning of the number in the bracket after rdd? 回答1: The number after RDD is its identifier: Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.3.0 /_/ Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_151) Type in expressions to have them evaluated. Type :help for more information. scala> val rdd = sc.range(0, 42) rdd: org.apache.spark.rdd.RDD[Long] = MapPartitionsRDD[1] at range at <console>:24 scala> rdd.id res0

问题 I have a question regarding the usage of local variables in closures when accessing Spark RDDs. The problem I would like to solve looks as follows: I have a list of textfiles that should be read into an RDD. However, first I need to add additional information to an RDD that is created from a single textfile. This additional information is extracted from the filename. Then, the RDDs are put into one big RDD using union(). from pyspark import SparkConf, SparkContext spark_conf = SparkConf()

问题 Can any one please correct my understanding on persisting by Spark. If we have performed a cache() on an RDD its value is cached only on those nodes where actually RDD was computed initially. Meaning, If there is a cluster of 100 Nodes, and RDD is computed in partitions of first and second nodes. If we cached this RDD, then Spark is going to cache its value only in first or second worker nodes. So when this Spark application is trying to use this RDD in later stages, then Spark driver has to

问题 My data are available as sets of Python 3 pickled files. Most of them are serialization of Pandas DataFrames . I'd like to start using Spark because I need more memory and CPU that one computer can have. Also, I'll use HDFS for distributed storage. As a beginner, I didn't found relevant information explaining how to use pickle files as input file. Does it exists? If not, are there any workaround? Thanks a lot 回答1: A lot depends on the data itself. Generally speaking Spark doesn't perform

问题 My goal is to group objects based on time overlap. Each object in my rdd contains a start_time and end_time . I'm probably going about this inefficiently but what I'm planning on doing is assigning an overlap id to each object based on if it has any time overlap with any of the other objects. I have the logic for time overlap down. Then, I hope to group by that overlap_id . So first, mapped_rdd = rdd.map(assign_overlap_id) final_rdd = mapped_rdd.reduceByKey(combine_objects) Now this comes to

问题 Given an input rdd or form 1: 6 7 2: 5 How can i get another rdd of form 1 6 1 7 2 5 and so on.. fails with message unicode item does not have attribute flatMap def get_str(x,y): ..code to flatmap return op text = sc.textFile(inputs) res = text.map(lambda l:l.split(":")).map(lambda (x,y):get_str(x,y)) 回答1: I'm not really into Python, but it looks like you're trying to use flatMap inside your map , but rather you need to replace your map with flatMap . In Scala, I would do: val text = sc

问题 I read in a DataFrame with a huge file holding on each line of it a JSON object as follows: { "userId": "12345", "vars": { "test_group": "group1", "brand": "xband" }, "modules": [ { "id": "New" }, { "id": "Default" }, { "id": "BestValue" }, { "id": "Rating" }, { "id": "DeliveryMin" }, { "id": "Distance" } ] } How could I manipulate in such way the DataFrame, to keep only the module with id="Default" ? How to just delete all the other, if id does not equal "Default" ? 回答1: As you said you have

问题 This question already has an answer here : Converting multiple different columns to Map column with Spark Dataframe scala (1 answer) Closed 2 years ago . I have dataframe as below and Appreciate if someone can help me to get the output in below different format. Input: |customerId|transHeader|transLine| |1001 |1001aa |1001aa1 | |1001 |1001aa |1001aa2 | |1001 |1001aa |1001aa3 | |1001 |1001aa |1001aa4 | |1002 |1002bb |1002bb1 | |1002 |1002bb |1002bb2 | |1002 |1002bb |1002bb3 | |1002 |1002bb

问题 I am new to Spark & scala. I have a requirement to process number of json files say from s3 location. These data is basically batch data which would be kept for reproccessing sometime later. Now my spark job should process these files in such a way that it should pick 5 raw json records and should send a message to Kafka topic. The reason of picking only 5 records is kafka topic is processing both real time and batch data simultaneously on the same topic. so the batch processing should not

问题 I'm trying to convert an RDD that has a fixed size lists of strings (a result of parsing CSV file) into and RDD of Rows. This is so I can turn it into a dataframe, because I need it into a dataframe to write to parquet. Anyway the only part I need help with is the converting of Rdd from list of strings to Row. The RDD variable name is RDD 回答1: I used: import org.apache.spark.sql._ val RowRDD = RDD.map(r => Row.fromSeq(r)) 来源： https://stackoverflow.com/questions/35441126/rdd-of-list-string