How come putting the GCM authentication tag at the end of a cipher stream require internal buffering during decryption?

Question

In Java, the \"default\" AES/GCM provider SunJCE will - during the decryption process - internally buffer 1) encrypted bytes used as input or 2) decrypted bytes

Answer 1

The short answer is that update() can't distinguish the ciphertext from the tag. The final() function can.

The long answer: Since Sun's specification requires the tag to be appended to the ciphertext, the tag needs to be stripped from the source buffer (ciphertext) during (or rather, prior to) decryption. However, because the ciphertext can be provided over the course of several update() calls, Sun's code does not know when to pull off the tag (in the context of update()). The last update() call does not know that it is the last update() call.

By waiting until the final() to actually do any crypto, it knows the full ciphertext + tag has been provided, and it can easily strip the tag off the end, given the tag length (which is provided in the parameter spec). It can't do crypto during the update because it would either treat some ciphertext as the tag or vice versa.

Basically, this is the drawback to simply appending the tag to the ciphertext. Most other implementations (e.g. OpenSSL) will provide the ciphertext and tag as separate outputs (final() returns the ciphertext, some other get() function returns the tag). Sun no doubt chose to do it this way in order to make GCM fit with their API (and not require special GCM-specific code from developers).

The reason encryption is more straightforward is that it has no need to modify its input (plaintext) like decryption does. It simply takes all data as plaintext. During the final, the tag is easily appended to the ciphertext output.

What @blaze said regarding protecting you from yourself is a possible rational, but it is not true that nothing can be returned until all ciphertext is known. Only a single block of ciphertext is needed (OpenSSL, for example, will give it to you). Sun's implementation only waits because it cannot know that that first block of ciphertext is just the first block of ciphertext. For all it knows, you're encrypting less than a block (requiring padding) and providing the tag all at once. Of course, even if it did give you the plaintext incrementally, you could not be sure of authenticity until the final(). All ciphertext is required for that.

There are, of course, any number of ways Sun could have made this work. Passing and retrieving the tag through special functions, requiring the length of the ciphertext during init(), or requiring the tag to be passed through on the final() call would all work. But, like I said, they probably wanted to make the usage as close to the other Cipher implementations as possible and maintain API uniformity.

Answer 2

I don't know why, but the current implementation writes every encoded byte you throw at it into a buffer until doFinal(), no matter what you do.

Source can be found here: GaloisCounterMode.java

This method is called from update and is given the bytes (+buffered ones) and is supposed to decrypt in case it can.

int decrypt(byte[] in, int inOfs, int len, byte[] out, int outOfs) {
    processAAD();

    if (len > 0) {
        // store internally until decryptFinal is called because
        // spec mentioned that only return recovered data after tag
        // is successfully verified
        ibuffer.write(in, inOfs, len);
    }
    return 0;
}

but it simply adds the data to ibuffer (ByteArrayOutputStream) and returns 0 as number of decrypted bytes. It does the whole decryption in doFinal then.

Given that implementation your only choices are to avoid that encryption or to manually build blocks of data you know your server can handle. There is no way to provide the tag data in advance and make it behave nicer.

Answer 3

Until entire ciphertext is known, algorithm can't tell if it was correct or tampered with. No decrypted bytes can be returned to use before decryption and authentication are completed.

Ciphertext buffering may be caused by the reasons @NameSpace mentioned, but plaintext buffering is here to not to let you shoot into your own leg.

Your best option is to encrypt data in small chunks. And don't forget to change nonce value between them.