rdd | 易学教程

RDD&Dataset&DataFrame

阅读更多关于 RDD&Dataset&DataFrame

Dataset创建 object DatasetCreation { def main(args: Array[String]): Unit = { val spark = SparkSession .builder() .appName("SparkSessionTest") .getOrCreate() import spark.implicits._ //1: range val ds1 = spark.range(0, 10, 2, 2) ds1.show() val dogs = Seq(Dog("jitty", "red"), Dog("mytty", "yellow")) val cats = Seq(new Cat("jitty", 2), new Cat("mytty", 4)) //2: 从Seq[T]中创建 val data = dogs val ds = spark.createDataset(data) ds.show() //3: 从RDD[T]中创建 val dogRDD = spark.sparkContext.parallelize(dogs) val dogDS = spark.createDataset(dogRDD) dogDS.show() val catRDD = spark.sparkContext.parallelize(cats)

Spark throws java.io.IOException: Failed to rename when saving part-xxxxx.gz

阅读更多关于 Spark throws java.io.IOException: Failed to rename when saving part-xxxxx.gz

问题 New Spark user here. I'm extracting features from many .tif images stored on AWS S3, each with identifier like 02_R4_C7. I'm using Spark 2.2.1 and hadoop 2.7.2. I'm using all default configurations like so: conf = SparkConf().setAppName("Feature Extraction") sc = SparkContext(conf=conf) sc.setLogLevel("ERROR") sqlContext = SQLContext(sc) And here is the function call that this fails on after some features are successfully saved in an image id folder as part-xxxx.gz files: features_labels_rdd

Spark throws java.io.IOException: Failed to rename when saving part-xxxxx.gz

阅读更多关于 Spark throws java.io.IOException: Failed to rename when saving part-xxxxx.gz

Huge memory consumption in Map Task in Spark

阅读更多关于 Huge memory consumption in Map Task in Spark

问题 I have a lot of files that contain roughly 60.000.000 lines. All of my files are formatted in the format {timestamp}#{producer}#{messageId}#{data_bytes}\n I walk through my files one by one and also want to build one output file per input file. Because some of the lines depend on previous lines, I grouped them by their producer. Whenever a line depends on one or more previous lines, their producer is always the same. After grouping up all of the lines, I give them to my Java parser. The

reduce() vs. fold() in Apache Spark

阅读更多关于 reduce() vs. fold() in Apache Spark

问题 What is the difference between reduce vs. fold with respect to their technical implementation? I understand that they differ by their signature as fold accepts additional parameter (i.e. initial value) which gets added to each partition output. Can someone tell about use case for these two actions? Which would perform better in which scenario consider 0 is used for fold ? Thanks in advance. 回答1: There is no practical difference when it comes to performance whatsoever: RDD.fold action is using

Remove Empty Partitions from Spark RDD

阅读更多关于 Remove Empty Partitions from Spark RDD

问题 I am fetching data from HDFS and storing it in a Spark RDD. Spark creates the number of partitions based on the number of HDFS blocks. This leads to a large number of empty partitions which also get processed during piping. To remove this overhead, I want to filter out all the empty partitions from the RDD. I am aware of coalesce and repartition, but there is no guarantee that all the empty partitions will be removed. Is there any other way to go about this? 回答1: There isn't an easy way to

Spark的存储级别

阅读更多关于 Spark的存储级别

Spark 中一个很重要的能力是将数据持久化（或称为缓存），在多个操作间都可以访问这些持久化的数据。当持久化一个 RDD 时，每个节点的其它分区都可以使用 RDD 在内存中进行计算，在该数据上的其他 action 操作将直接使用内存中的数据。这样会让以后的 action 操作计算速度加快（通常运行速度会加速 10 倍）。缓存是迭代算法和快速的交互式使用的重要工具。 RDD 可以使用 persist() 方法或 cache() 方法进行持久化。数据将会在第一次 action 操作时进行计算，并缓存在节点的内存中。Spark 的缓存具有容错机制，如果一个缓存的 RDD 的某个分区丢失了，Spark 将按照原来的计算过程，自动重新计算并进行缓存。另外，每个持久化的 RDD 可以使用不同的存储级别进行缓存，例如，持久化到磁盘、已序列化的 Java 对象形式持久化到内存（可以节省空间）、跨节点间复制、以 off-heap 的方式存储在 Tachyon。这些存储级别通过传递一个 StorageLevel 对象（Scala、Java、Python）给 persist() 方法进行设置。cache() 方法是使用默认存储级别的快捷设置方法，默认的存储级别是 StorageLevel.MEMORY_ONLY（将反序列化的对象存储到内存中）。详细的存储级别介绍如下 : MEMORY_ONLY : 将

passing value of RDD to another RDD as variable - Spark #Pyspark [duplicate]

阅读更多关于 passing value of RDD to another RDD as variable - Spark #Pyspark [duplicate]

问题 This question already has answers here : How to get a value from the Row object in Spark Dataframe? (3 answers) Closed last year . I am currently exploring how to call big hql files (contains 100 line of an insert into select statement) via sqlContext. Another thing is, The hqls files are parameterize, so while calling it from sqlContext, I want to pass the parameters as well. Have gone through loads of blogs and posts, but not found any answers to this. Another thing I was trying, to store

Pyspark Merge WrappedArrays Within a Dataframe

阅读更多关于 Pyspark Merge WrappedArrays Within a Dataframe

问题 The current Pyspark dataframe has this structure (a list of WrappedArrays for col2): +---+---------------------------------------------------------------------+ |id |col2 | +---+---------------------------------------------------------------------+ |a |[WrappedArray(code2), WrappedArray(code1, code3)] | +---+---------------------------------------------------------------------+ |b |[WrappedArray(code5), WrappedArray(code6, code8)] | +---+-------------------------------------------------------

Convert an RDD to iterable: PySpark?

阅读更多关于 Convert an RDD to iterable: PySpark?

问题 I have an RDD which I am creating by loading a text file and preprocessing it. I dont want to collect it and save it to the disk or memory(entire data) but rather want to pass it to some other function in python which consumes data one after the other is form of iterable. How is this possible? data = sc.textFile('file.txt').map(lambda x: some_func(x)) an_iterable = data. ## what should I do here to make it give me one element at a time? def model1(an_iterable): for i in an_iterable: do_that(i