why i cant change the property of nodes using map function in Spark?

问题

i am working with GraphX in Spark to process a graph. i have a val common_neighbors: RDD[VertexId] that holds some vertexId. i use map function to transform it into a structure such as (node,1), which node is the ID of the vertex and 1 is its initial property. the code for transforming is below:

val p =common_neighbors.map(x=>(x,1))

i have a graph that has a structure such as: (node,node_property(label,isDefined)). for example (1,(14,true)). this means node with ID=1 has label=14 and isDefined.

i want to transform the nodes property in p in a parallel and distributed way if the node's label in the graph is higher than 5. the code is below:

val x=p.map(node=>{


        val temp_property=graph.vertices.filter(x=>x._1==node._1).values.take(1)
        if(temp_property(0).label > 5) {

          (node._1,((temp_property(0).label)+5))
        }
        })

But when i execute the code, i get Errors. what is the problem with this?how can i fix this?

回答1:

as i understood your problem, you are using a RDD inside another RDD, that its not true. instead of that you can do this: you can join your common_neighbors with the real graph by using below code:

val new_val=common_neighbors.join(work_graph.vertices)

then through some map transformation you can make the structure of your graph in the new_val and then you can use map or mapValues to do every operation on values of the new_val

来源：https://stackoverflow.com/questions/60522369/why-i-cant-change-the-property-of-nodes-using-map-function-in-spark