Spark : DB connection per Spark RDD partition and do mapPartition

后端 未结 2 805
野趣味
野趣味 2020-12-05 22:21

I want to do a mapPartitions on my spark rdd,

    val newRd = myRdd.mapPartitions(
      partition => {

        val connection = new DbConnection /*crea         


        
2条回答
  •  忘掉有多难
    2020-12-05 22:57

    rdd.foreachPartitionAsync(iterator->{
    
    // this object will be cached inside each executor JVM. For the first time, the //connection will be created and hence forward, it will be reused. 
    // Very useful for streaming apps
    DBConn conn=DBConn.getConnection()
    while(iterator.hasNext()) {
      conn.read();
    }
    
    });
    
    public class DBConn{
    private static dbObj=null;
    
    //Create a singleton method that returns only one instance of this object
    }
    
    }
    

提交回复
热议问题