Looping over byte channels

问题

While reading from the good-old InputStream, I used the following code(with which I was never comfortable) :

    int read = 0;
    InputStream is = ....;

    while((i = is.read() != -1){

        ....
    }

Now I'm trying to read 10MB from an InputStream using NIO :

        protected void doPost(HttpServletRequest request,
            HttpServletResponse response) throws ServletException, IOException {
        // TODO Auto-generated method stub

        System.out.println("In Controller.doPost(...)");

        ByteBuffer chunk = ByteBuffer.allocateDirect(1000000);

        /* Source channel */
        int numRead = 0;
        ReadableByteChannel rbc = Channels.newChannel(request.getInputStream());

        /* Destination channel */
        File destFile = new File(
                "D:\\SegyDest.sgy");
        FileOutputStream destFileFos = new FileOutputStream(destFile);
        FileChannel destFileChannel = destFileFos.getChannel();

        /* Read-Write code */
        while (numRead >= 0) {
            chunk.rewind();
            numRead = rbc.read(chunk);

            System.out.println("numRead = " + numRead);

            chunk.rewind();
            destFileChannel.write(chunk);
        }

        /* clean-up */
        rbc.close();
        destFileChannel.close();
        destFileFos.close();

        request.setAttribute("ops", "File Upload");
        request.getRequestDispatcher("/jsp/Result.jsp").forward(request,
                response);
    }

My question is /* How to loop over the source channel to read all the bytes ? */

回答1:

OR perform IO in chunks of more than 1 byte the API like so:

byte[] bA = new byte[4096];
int i;
InputStream is = ....;
OutputStream os = ....;

while((i = is.read(bA) != -1){
    os.write(bA, 0, i);
}

I've looked at your other question and my comments still stand. NIO is not the solution you are looking for. You have a low end machine with limits RAM acting as a proxy.

The best you can do is have your Servlet create a new thread, have this thread create and setup an outgoing connection using NIO sockets/HTTP-libraries. This new (and extra) thread is waiting on any of 3 things to happen and it pushes whatever APIs to try and make progress in these 3 areas.

The 3 things are:

Trying to write data to the remote server (if there is buffered in memory data to send)
Waiting for the main Servlet thread to indicate there is new data in the shared buffer. Or that End-of-stream was reached.
Waiting for the main Servlet thread to indicate the extra thread needs to shutdown (this is error recovery and cleanup).

You probably need a drainWithTimeout(long millis) function that the doPost() method calls on the extra thread to give it an amount of time to push the final data to the remote server. This gets called when an End-of-Stream if observed by the Servlet from the InputStream.

You MUST ensure your extra thread is 100% reliably reaped before the doPost() method returns. So controlling startup/shutdown of it is important, especially in the scenarios that the InputStream had an error because the sending client disconnected or was idle too long.

Then two threads (the normal Servlet thread in doPost() and the new thread you create) would setup and share some arbitrary memory buffer, maybe 16Mb or more that is shared.

If you can not have a 16Mb buffer due to limitations in clients/concurrent-users and 2Gb RAM then you really should stick with the example code at the top of this answer, since the network and the O/S kernels will already buffer some Mb's of data.

The point of using two threads is that you can not fix the issue that the Servlet API receiving the data is a blocking I/O API, you can not change that if you are writing the application to conform to Servlet specification/standards. If you know your specific Servlet container has a feature then that is outside the scope of this answer.

The two threads allow the main Servlet doPost thread to be in control and STILL use a blocking I/O API for InputStream.

There is no point using one thread and a blocking InputStream with a non-blocking OutputStream, you still have the problem that you can not service the output stream while the in.read() API call is blocked (waiting for more data or End-of-stream).

回答2:

The correct way to copy between NIO channels is as follows:

while (in.read(buffer) > 0 || buffer.position() > 0)
{
  buffer.flip();
  out.write(buffer);
  buffer.compact();
}

Note that this automatically takes care of EOS, partial reads, and partial writes.

来源：https://stackoverflow.com/questions/12507233/looping-over-byte-channels

标签

nio

java-io