Merging multiple files into one within Hadoop

前端 未结 8 880
遇见更好的自我
遇见更好的自我 2020-12-01 02:18

I get multiple small files into my input directory which I want to merge into a single file without using the local file system or writing mapreds. Is there a way I could do

8条回答
  •  广开言路
    2020-12-01 02:50

    All the solutions are equivalent to doing a

    hadoop fs -cat [dir]/* > tmp_local_file  
    hadoop fs -copyFromLocal tmp_local_file 
    

    it only means that the local m/c I/O is on the critical path of data transfer.

提交回复
热议问题