How to read files with an offset from Hadoop using Java

问题

Problem: I want to read a section of a file from HDFS and return it, such as lines 101-120 from a file of 1000 lines.

I don't want to use seek because I have read that it is expensive.

I have log files which I am using PIG to process down into meaningful sets of data. I've been writing an API to return the data for consumption and display by a front end. Those processed data sets can be large enough that I don't want to read the entire file out of Hadoop in one slurp to save wire time and bandwidth. (Let's say 5 - 10MB)

Currently I am using a BufferedReader to return small summary files which is working fine

ArrayList lines = new ArrayList();
...
for (FileStatus item: items) {
    // ignoring files like _SUCCESS
    if(item.getPath().getName().startsWith("_")) {
        continue;
    }           

    in = fs.open(item.getPath());
    BufferedReader br = new BufferedReader(new InputStreamReader(in));

    String line;
    line = br.readLine();
    while (line != null) {
        line = line.replaceAll("(\\r|\\n)", "");
        lines.add(line.split("\t"));
        line = br.readLine();
    }
}

I've poked around the interwebs quite a bit as well as Stack but haven't found exactly what I need.

Perhaps this is completely the wrong way to go about doing it and I need a completely separate set of code and different functions to manage this. Open to any suggestions.

Thanks!

As added noted based on research from the below discussions: How does Hadoop process records records split across block boundaries? Hadoop FileSplit Reading

回答1:

I think SEEK is a best option for reading files with huge volumes. It did not cause any problems to me as the volume of data that i was reading was in the range of 2 - 3GB. I did not encounter any issues till today but we did use file splitting to handle the large data set. below is the code which you can use for reading purpose and test the same.

public class HDFSClientTesting {

/**
 * @param args
 */
public static void main(String[] args) {
    // TODO Auto-generated method stub

  try{

 //System.loadLibrary("libhadoop.so");
    Configuration conf = new Configuration();
    FileSystem fs = FileSystem.get(conf);
    conf.addResource(new Path("core-site.xml"));


    String Filename = "/dir/00000027";
    long ByteOffset = 3185041;



    SequenceFile.Reader rdr = new SequenceFile.Reader(fs, new Path(Filename), conf);
    Text key = new Text();
    Text value = new Text();

    rdr.seek(ByteOffset);
    rdr.next(key,value);
    //Plain text
    JSONObject jso = new JSONObject(value.toString());
    String content = jso.getString("body");
    System.out.println("\n\n\n" + content + "\n\n\n");

    File file =new File("test.gz");
    file.createNewFile();

}
  catch (Exception e ){
    throw new RuntimeException(e);

}
 finally{

 } 

  }

}

来源：https://stackoverflow.com/questions/24064178/how-to-read-files-with-an-offset-from-hadoop-using-java

标签

java

Hadoop

iteration

bufferedreader

offset