The following can be found on various forums in relation to mapPartitions and map:
In my opinion, connection should be kept out and created just once before map and closed post task completion.
val connection = new DbConnection /creates a db connection per partition/
val newRd = myRdd.mapPartitions(
partition => {
val newPartition = partition.map(
record => {
readMatchingFromDB(record, connection)
})
newPartition
})
connection.close()