Spark Streaming: Could not compute split, block not found

Question

I am trying to use Spark Streaming with Kafka (version 1.1.0) but the Spark job keeps crashing due to this error:

14/11/21 12:39:23 ERROR TaskSetManager: Tas         


        

Answer 1

Check the following.

1) Did you create the streaming context properly as in

def functionToCreateContext(): StreamingContext = {
    val ssc = new StreamingContext(...)   // new context
    val lines = ssc.socketTextStream(...) // create DStreams
    ...
    ssc.checkpoint(checkpointDirectory)   // set checkpoint directory
    ssc
}

// Get StreamingContext from checkpoint data or create a new one
val context = StreamingContext.getOrCreate(checkpointDirectory, functionToCreateContext _)

// Do additional setup on context that needs to be done,
// irrespective of whether it is being started or restarted
context. ...

// Start the context
context.start()
context.awaitTermination()

Your initialization is incorrect.

Have a look at the below

Eg : code at recoverableNetworkCount App

2) Have you enabled the property write ahead log "spark.streaming.receiver.writeAheadLog.enable"

3) Check the stability of streaming in the Streaming UI. processing time < batch interval.