Number of Cycles from list of values, which are mix of positives and negatives in Spark and Scala

问题

Have an RDD with List of values, which are mix of positives and negatives. Need to compute number of cycles from this data.

For example,

val range = List(sampleRange(2020,2030,2040,2050,-1000,-1010,-1020,Starting point,-1030,2040,-1020,2050,2040,2020,end point,-1060,-1030,-1010)

the interval between each value in above list is 1 second. ie., 2020 and 2030 are recorded in 1 second interval and so on.

how many times it turns from negative to positive and stays positive for >= 2 seconds.
If >= 2 seconds it is a cycle.

Number of cycles: Logic
Example 1: List(1,2,3,4,5,6,-15,-66)
No. of cycles is 1.
Reason: As we move from 1st element of list to 6th element, we had 5 intervals which means 5 seconds. So one cycle. As we move to 6th element of list, it is a negative value. So we start counting from 6th element and move to 7th element. The negative values are only 2 and interval is only 1. So not counted as cycle.
Example 2:
List(11,22,33,-25,-36,-43,20,25,28)
No. of cycles is 3.
Reason: As we move from 1st element of list to 3rd element, we had 2 intervals which means 2 seconds. So one cycle As we move to 4th element of list, it is a negative value. So we start counting from 4th element and move to 5th, 6th element. we had 2 intervals which means 2 seconds. So one cycle As we move to 7th element of list, it is a positive value. So we start counting from 7th element and move to 8th, 9th element. we had 2 intervals which means 2 seconds. So one cycle.

range is a RDD in the use case. It looks like
scala> range
range: Seq[com.Range] = List(XtreamRange(858,890,899,920,StartEngage,-758,-790,-890,-720,920,940,950))

回答1:

You can encode this "how many times it turns from negative to positive and stays positive for >= 2 seconds. If >= 2 seconds it is a cycle." pretty much directly into a pattern match with a guard. The expression if(h < 0 && ht > 0 && hht > 0) checks for a cycle and adds one to the result then continues with the rest of the list.

def countCycles(xs: List[Int]): Int = xs match {
 case Nil => 0
 case h::ht::hht::t if(h < 0 && ht > 0 && hht > 0) => 1 + countCycles(t)
 case h::t => countCycles(t)
}

scala> countCycles(range)
res7: Int = 1

回答2:

A one liner

range.sliding(3).count{case f::s::t::Nil => f < 0 && s > 0 && t > 0}

This generates all sub-sequences of length 3 and counts how many are -ve, +ve, +ve

Generalising cycle length

def countCycles(n:Int, xs:List[Int]) = xs.sliding(n+1)
                                         .count(ys => ys.head < 0 && ys.tail.forall(_ > 0))

回答3:

The below code would help you resolve you query.

object CycleCheck {
   def main(args: Array[String]) {
     var data3 = List(1, 4, 82, -2, -12, "startingpoint", -9, 32, 76,45, -98, 76, "Endpoint", -24)
     var data2 = data3.map(x => getInteger(x)).filter(_ != "unknown").map(_.toString.toInt)
     println(data2)
     var nCycle = findNCycle(data2)
     println(nCycle)
   }

   def getInteger(obj: Any) = obj match {
     case n: Int => obj
     case _     => "unknown"

   }
def findNCycle(obj: List[Int]) : Int = {
 var cycleCount =0
 var sign = ""
 var signCheck="+"
 var size = obj.size - 1
 var numberOfCycles=0
 var i=0
 for( x <- obj){
   if (x < 0){
     sign="-"
   }
   else if (x > 0){
     sign="+"
    }

  if(signCheck.equals(sign))
        cycleCount=cycleCount+1

  if(!signCheck.equals(sign) && cycleCount>1){
       cycleCount = 1
       numberOfCycles=numberOfCycles+1
  }    
  if(size==i && cycleCount>1)
      numberOfCycles= numberOfCycles+1


 if(cycleCount==1)
      signCheck = sign;

  i=i+1
 }

  return numberOfCycles

} }

来源：https://stackoverflow.com/questions/39219085/number-of-cycles-from-list-of-values-which-are-mix-of-positives-and-negatives-i

标签

scala

Hadoop

apache-spark

bigdata