Reading AES/GCM encoded data in chunks with BouncyCastle in Java

Question

I am trying to figure out how to read data that was encoded with AES/GCM/NoPadding. The data that I am working with will be arbitrarily large, and I'm hoping to read it in chunks, but I'm having difficulties figuring out how that would be accomplished. Here is an example of where I'm at right now:

@Test
public void chunkDecrypt() throws Exception {
    key = MessageDigest.getInstance("MD5").digest("som3C0o7p@s5".getBytes());
    iv = Hex.decode("EECE34808EF2A9ACE8DF72C9C475D751");
    byte[] ciphertext = Hex
            .decode("EF26839493BDA6DA6ABADD575262713171F825F2F477FDBB53029BEADB41928EA5FB46737D7A94D5BE74B6049008443664F0E0D883943D0EFBEA09DB");

    Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BC");
    cipher.init(Cipher.DECRYPT_MODE, new SecretKeySpec(key, "AES"), new IvParameterSpec(iv));

    byte[] fullDecryptedPlainText = cipher.doFinal(ciphertext);
    assertThat(new String(fullDecryptedPlainText),
            is("The quick brown fox jumps over the lazy dogs"));

    byte[] first32 = Arrays.copyOfRange(ciphertext, 0, 32);
    byte[] final28 = Arrays.copyOfRange(ciphertext, 32, 60);
    byte[] decryptedChunk = new byte[32];

    int num = cipher.update(first32, 0, 32, decryptedChunk);
    assertThat(num, is(16));
    assertThat(new String(decryptedChunk, 0, 16), is("The quick brown "));

    num = cipher.update(first32, 0, 32, decryptedChunk);
    assertThat(num, is(32));
    assertThat(new String(decryptedChunk, 0, 16), is("fox jumps over t"));

    num = cipher.update(final28, 0, 24, decryptedChunk);
    assertThat(num, is(44));
    assertThat(new String(decryptedChunk, 0, 12), is("he lazy dogs"));
}

Note that I get past the first assert no problem, so the data can be decoded in a single go. Also, the next two sets of asserts (decoding the first 32 bytes in 16-byte chunks) work "correctly" but I arrived at this formula through trial-and-error. There are a few things about them that I don't understand:

• Even though I am reading in 16-byte chunks, all of my numbers seem to need to be multiples of 32. If I change to the following code, then the first call to cipher.update() fails with a return value of 0.

  byte[] first16 = Arrays.copyOfRange(ciphertext, 0, 16);
  byte[] decryptedChunk = new byte[16];
  
  int num = cipher.update(first16, 0, 16, decryptedChunk);
  

• If I change back to 32 on the input side, but I work with a 16-byte output buffer, then the first call succeeds and returns the expected data, but the second call to cipher.update() throws ArrayIndexOutOfBoundsException.

  byte[] first32 = Arrays.copyOfRange(ciphertext, 0, 32);
  byte[] decryptedChunk = new byte[16];
  
  int num = cipher.update(first32, 0, 32, decryptedChunk);
  num = cipher.update(first32, 0, 32, decryptedChunk);
  

• So, if I change the code back to my original example (decryptedChunk is sized at 32 bytes) then the third call to cipher.update() returns a value of 16 (meaning what???) and decryptedChunk contains garbage data.

• I also tried replacing the last call to cipher.update() with a call to cipher.doFinal() instead:

  decryptedChunk = cipher.doFinal(final28);
  assertThat(new String(decryptedChunk, 0, 12), is("he lazy dogs"));
  

But this fails with a BadPaddingException (mac check in GCM failed).

Any suggestions?

---

Update with solution

After playing around some with the suggested code from Ebbe M. Pedersen, I have been able to put together the following solution:

@Test
public void chunkDecrypt() throws Exception {
    byte[] key = MessageDigest.getInstance("MD5").digest("som3C0o7p@s5".getBytes());
    byte[] iv = Hex.decode("EECE34808EF2A9ACE8DF72C9C475D751");
    byte[] ciphertext = Hex
            .decode("EF26839493BDA6DA6ABADD575262713171F825F2F477FDBB53029BEADB41928EA5FB46737D7A94D5BE74B6049008443664F0E0D883943D0EFBEA09DB");

    Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BC");
    cipher.init(Cipher.DECRYPT_MODE, new SecretKeySpec(key, "AES"), new IvParameterSpec(iv));

    int chunkSize = 16;
    byte[] inBuffer = new byte[chunkSize];
    int outBufferSize = ((chunkSize + 15) / 16) * 16;
    byte[] outBuffer = new byte[outBufferSize];

    for (int i = 0; i < ciphertext.length; i += chunkSize) {
        int thisChunkSize = Math.min(chunkSize, ciphertext.length - i);
        System.arraycopy(ciphertext, i, inBuffer, 0, thisChunkSize);
        int num = cipher.update(inBuffer, 0, thisChunkSize, outBuffer);
        if (num > 0) {
            logger.debug("update #" + ((i / chunkSize) + 1) + " - data <"
                    + new String(outBuffer, 0, num) + ">");
        }
    }
    int num = cipher.doFinal(inBuffer, chunkSize, 0, outBuffer);
    logger.debug("doFinal - data <" + new String(outBuffer, 0, num) + ">");
}

This works properly for any value of chunkSize that I have selected. I have marked that answer as accepted. Thank you all for the help.

Answer 1

Block ciphers [ed: in Bouncy Castle] have an internal buffer that they keep updating, and only when they have enough data for a full block, will the decrypt occur, and a chunk of the decrypted data be returned.

You can see this if you try and decrypt it 1 byte at a time like this:

    byte[] buffer = new byte[32];
    for (int i = 0; i < ciphertext.length; i++) {
        int num = cipher.update(ciphertext, i, 1, buffer);
        if (num > 0) {
            System.out.println("update #" + (i + 1) + " - data <" + new String(buffer, 0, num) + ">");
        }
    }
    int num = cipher.doFinal(ciphertext, ciphertext.length, 0, buffer);
    System.out.println("doFinal - data <" + new String(buffer, 0, num) + ">");

This gives the following output with your encrypted data:

update #32 - data <The quick brown >
update #48 - data <fox jumps over t>
doFinal - data <he lazy dogs>

Notice that I need to do a doFinal(), to get the last piece of data out.

---

Note that this is particular to the Bouncy Castle implementation, at least up to version 1.50. CTR mode allows to pre-compute blocks of the key stream used to encrypt/decrypt the data (by XOR'ing, analog of a OTP encryption). So in principle each byte or even bit can be encrypted/decrypted on its own.