Copying files from a hdfs directory to another with oozie distcp-action

Question

My actions

start_fair_usage ends with status okey, but test_copy returns

Main class [org.apache.oozie.action.hadoop.DistcpM         


        

Answer 1

Here is what I did in the end.

  

  
    
    
    
  

  
    
      
        
      
      ${NAME_NODE}${srcFolder}/*mta.gz
      ${NAME_NODE}${dstFolder}mta/
    
    
    
  

  
    
      
        
      
      ${NAME_NODE}${srcFolder}/*rcr.gz
      ${NAME_NODE}${dstFolder}rcr/
    
    
    
  

  
    
      
        
      
      ${NAME_NODE}${srcFolder}/*sub.gz
      ${NAME_NODE}${dstFolder}sub/
    
    
    
  

  

  
    Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
  
  

Turned out it was possible to use wildcards in distcp, so I didn't need bash at all.

Also. Some people adviced me to write it in scala.

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path, FileUtil}

val conf = new Configuration()
val fs = FileSystem.get(conf)

val listOfFileTypes = List("mta", "rcr", "sub")
val listOfPlatforms = List("B", "C", "H", "M", "Y")

for(fileType <- listOfFileTypes){
  FileUtil.fullyDeleteContents(new File("/apps/hive/warehouse/arstel.db/fair_usage/fct_evkuzmin/file_" + fileType))
  for (platform <- listOfPlatforms) {
    var srcPaths = fs.globStatus(new Path("/user/comverse/data/" + "20170404" + "_" + platform + "/*" + fileType + ".gz"))
    var dstPath = new Path("/apps/hive/warehouse/arstel.db/fair_usage/fct_evkuzmin/file_" + fileType)

    for(srcPath <- srcPaths){
      println("copying " + srcPath.getPath.toString)
      FileUtil.copy(fs, srcPath.getPath, fs, dstPath, false, conf)
    }
  }
}

Both things work, thought I haven't tried to run the scala script in Oozie.