Best way to Pipe InputStream to OutputStream [duplicate]

Question

This question already has an answer here:

• Easy way to write contents of a Java InputStream to an OutputStream 22 answers

I was to trying to find the best way to pipe the InputStream to OutputStream. I don't have an option to use any other libraries like Apache IO. Here is the snippet and output.

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.channels.FileChannel;

public class Pipe {
    public static void main(String[] args) throws Exception {

        for(PipeTestCase testCase : testCases) {
            System.out.println(testCase.getApproach());
            InputStream is = new FileInputStream("D:\\in\\lft_.txt");
            OutputStream os = new FileOutputStream("D:\\in\\out.txt");

            long start = System.currentTimeMillis();            
            testCase.pipe(is, os);
            long end = System.currentTimeMillis();

            System.out.println("Execution Time = " + (end - start) + " millis");
            System.out.println("============================================");

            is.close();
            os.close();
        }

    }

    private static PipeTestCase[] testCases = {

        new PipeTestCase("Fixed Buffer Read") {         
            @Override
            public void pipe(InputStream is, OutputStream os) throws IOException {
                byte[] buffer = new byte[1024];
                while(is.read(buffer) > -1) {
                    os.write(buffer);   
                }
            }
        },

        new PipeTestCase("dynamic Buffer Read") {           
            @Override
            public void pipe(InputStream is, OutputStream os) throws IOException {
                byte[] buffer = new byte[is.available()];
                while(is.read(buffer) > -1) {
                    os.write(buffer);   
                    buffer = new byte[is.available() + 1];
                }
            }
        },

        new PipeTestCase("Byte Read") {         
            @Override
            public void pipe(InputStream is, OutputStream os) throws IOException {
                int c; 
                while((c = is.read()) > -1) {
                    os.write(c);    
                }
            }
        }, 

        new PipeTestCase("NIO Read") {          
            @Override
            public void pipe(InputStream is, OutputStream os) throws IOException {
                FileChannel source      = ((FileInputStream) is).getChannel(); 
                FileChannel destnation  = ((FileOutputStream) os).getChannel();
                destnation.transferFrom(source, 0, source.size());
            }
        }, 

    };
}


abstract class PipeTestCase {
    private String approach; 
    public PipeTestCase( final String approach) {
        this.approach = approach;           
    }

    public String getApproach() {
        return approach;
    }

    public abstract void pipe(InputStream is, OutputStream os) throws IOException;
}

Output (~4MB input file) :

Fixed Buffer Read
Execution Time = 71 millis
============================================
dynamic Buffer Read
Execution Time = 167 millis
============================================
Byte Read
Execution Time = 29124 millis
============================================
NIO Read
Execution Time = 125 millis
============================================

'Dynamic Buffer Read' uses available() method. But it is not reliable as per java docs

It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream.

'Byte Read' seems to be very slow.

So 'Fixed Buffer Read' is the best option for pipe? Any thoughts?

Answer 1

I would say a fixed buffer size is the best/easiest to understand. However there are a few problems.

• You're writing the entire buffer to the output stream each time. For the final block the read may have read < 1024 bytes so you need to take this into account when doing the write (basically only write number of bytes returned by read()

• In the dynamic buffer case you use available(). This is not a terribly reliable API call. I'm not sure in this case inside a loop whether it will be ok, but I wouldn't be suprised if it was implemented sub-optimally in some implementations of InputStream.

• The last case you are casting to FileInputStream. If you intend for this to be general purpose then you can't use this approach.

Answer 2

I came across this, and the final read can cause problems.

SUGGESTED CHANGE:

public void pipe(InputStream is, OutputStream os) throws IOException {
  int n;
  byte[] buffer = new byte[1024];
  while((n = is.read(buffer)) > -1) {
    os.write(buffer, 0, n);   // Don't allow any extra bytes to creep in, final write
  }
 os.close ();

I also agree that 16384 is probably a better fixed buffer size than 1024.

IMHO...

Answer 3

For people looking for an one-liner, here's a solution from apache commons:

IOUtils.copy(inputStream, outputStream);

Documentation here. There are multiple copy methods with different parameters. It is also possible to specify the buffer size.

Answer 4

java.io contains PipedInputStream and PipedOutputStream

PipedInputStream input = new PipedInputStream();
PipedOutputStream output = new PipedOutputStream (input);

write to input and it will be visible in output as an Outputstream. Things can work the other way around as well