Hadoop java.io.IOException: Mkdirs failed to create /some/path

Question

When I try to run my Job I am getting the following exception:

Exception in thread \"main\" java.io.IOException: Mkdirs failed to create /some/path
    at or         


        

Answer 1

I ran into this same issue while building MapReduce jobs on a Mac with MacOS Sierra. The same code runs without problems on Ubuntu Linux (14.04 LTS and 16.04 LTS). MapReduce distribution was 2.7.3, and was configured for Single Node, standalone operation. The problem appears to be related to copying license files into a META_INF directory. My problem was solved by adding a transformer into the Maven Shade plugin configuration, specifically: ApacheLicenseResourceTransformer.

Here is the relevant section of the POM.xml, which goes as part of the section:

                                                                                                             org.apache.maven.plugins                                                                      
   maven-shade-plugin                                                                      
   3.0.0                                                                                         
                                                                                                        
                                                                                                         
       package                                                                                       
                                                                                                             
         shade                                                                                         
                                                                                                            
                                                                                                     
                                                                                                      
                  
             path.to.your.main.class.goes.here                                        
                                                                                                      
             
                                                                                                      
                                                                                                     
                                                                                                    
                                                                                                        
                                                                                                       
   

Notice that I also use the ManifestResourceTransformerto specify the main class for the MapReduce Job.