distributed

I am developing a Spark processing framework which reads large CSV files, loads them into RDD's, performs some transformations and at the end saves some statistics. The CSV files in question are around 50GB on average. I'm using Spark 2.0. My question is: When I load the files using sparkContext.textFile() function, does the file needs to be stored in the memory of the driver first, and then it is distributed to the workers (thus requiring a rather large amount of memory on the driver)? Or the file is read "in parallel" by every worker, in a way none of them needs to store the whole file, and

In raft paper section 5.4.2 If a leader crashes before committing an entry, future leaders will attempt to finish replicating the entry. However, a leader cannot immediately conclude that an entry from a previous term is committed once it is stored on a majority of servers. There could be a situation where an old log entry is stored on a majority of servers, yet can still be overwritten by a future leader. The author mentioned that to avoid the situation above To eliminate problems like the one in Figure 8, Raft never commits log entries from previous terms by counting replicas. Only log

Does anyone know how to implement the Natural-Join operation between two datasets in Hadoop? More specifically, here's what I exactly need to do: I am having two sets of data: point information which is stored as (tile_number, point_id:point_info) , this is a 1:n key-value pairs. This means for every tile_number, there might be several point_id:point_info Line information which is stored as (tile_number, line_id:line_info) , this is again a 1:m key-value pairs and for every tile_number, there might be more than one line_id:line_info As you can see the tile_numbers are the same between the two