Changing an attribute in an object that belongs to RDD

问题

I have the following code :

      def generateStoriesnew(outputPath: String, groupedRDD:RDD[(String,Iterable[String])], isInChurnMode: Boolean, isInChurnPeriod: Boolean) {
val windowedRDD = groupedRDD.map(SOME CODE)
 var  windowedRDD2 = windowedRDD.filter(r => r != null).map(a=>a.churnPeriod(isInChurnPeriod,isInChurnMode))  
val prettyStringRDD = windowedRDD2.map(r => {
  r.toString
})
prettyStringRDD.saveAsTextFile(outputPath)

}

and here is the code for ChurnPriod function:

def churnPeriod( churnPeriod:Boolean, churnMode: Boolean): Unit = {  
if (churnMode && rootEventType.equalsIgnoreCase("c")){
  var churnCustStory: CustStoryN = null
  var nonChurnCustStory: CustStoryN = null
  var churnPeriodEventStory: mutable.MutableList[StoryEventN] = null
  var NonChurnEventstory: mutable.MutableList[StoryEventN] = null
  churnPeriodEventStory = new mutable.MutableList[StoryEventN]
  NonChurnEventstory = new mutable.MutableList[StoryEventN]     
  var lastEventChurnPeriod = true
  var currentEventStory = eventStory
  var max = currentEventStory.length
  println(max);
  if (currentEventStory.size > 0) {
    for (i <- 0 until currentEventStory.length) {
      var currentEvent = currentEventStory(i)
      if (currentEvent.timeSenseRootEvent < 90) {          
        churnPeriodEventStory.+=(currentEvent)
        //lastEventChurnPeriod = true
      }
      else {
        NonChurnEventstory.+=(currentEvent)
        lastEventChurnPeriod = false
      }
    }
  }
if (churnPeriod)
  eventStory = churnPeriodEventStory
else
  eventStory=null

} }

but churn period function does not change eventstory which is a member of a custstory class. what am I missing here ?

    class CustStoryN (val custId:String,
             var rootEventType:String,
             var rootEventTime:Long,
             var eventStory:mutable.MutableList[StoryEventN])

my hypothesis is either: 1.map is not the right transformation for the function that I have 2.churnPeriod function never get called 3.I can not change eventstory which is a member of cust story class Does anyone have any idea how I can troubleshoot this problem?

回答1:

This would be trivial to determine by debugging. Just put a few breakpoints and you can see if the program stops inside the function, and what transformations do occur.

My guess would be that the problem is this line: if (currentEventStory.size > 0), thus, a list that starts at size 0 remains at size 0 forever. Another option is that churnPeriod is never true, thus, you compute a lot but never assign to the eventStory variable.

Your code does need a good cleanup ;-)

来源：https://stackoverflow.com/questions/30246628/changing-an-attribute-in-an-object-that-belongs-to-rdd

标签

scala

class

rdd