Data read has a different length than the expected: dataLength=964481363; expectedLength=964481376;

Question

I am trying to read an encrypted Object from S3 and then upload it to a new bucket in S3 with a different encryption key. While I am trying to read the object, using getObject() and trying to put the object using PutObject request.

When I print the content length of the source, it is 964481376. So, where does the data Length comes from? which is 964481363

The metadata of my source file looks like this:

Content-Type

binary/octet-stream

x-amz-meta-x-amz-unencrypted-content-length

964481363

x-amz-meta-x-amz-wrap-alg

kms

I can't share the KMS key.

public boolean copyS3Object(AmazonS3Encryption sourceS3Client, AmazonS3URI sourceS3Uri,
            AmazonS3Encryption destS3Client, AmazonS3URI destS3Uri) throws AmazonServiceException, IOException {
        //String eTag;
        S3Object sourceS3Object = null;

        try {
            //Get the Source object stream
            sourceS3Object = sourceS3Client.getObject(sourceS3Uri.getBucket(), sourceS3Uri.getKey());
            ObjectMetadata objectMetadata = sourceS3Object.getObjectMetadata();
            System.out.println("content length " + objectMetadata.getContentLength());
            PutObjectRequest putRequest = new PutObjectRequest(destS3Uri.getBucket(), destS3Uri.getKey(),
                    sourceS3Object.getObjectContent(), sourceS3Object.getObjectMetadata());

            destS3Client.putObject(putRequest);
//
//            if (eTag.isEmpty()) {
//                System.out.println("Copy failed, New object in " + destS3Uri.toString() + " is empty");
//                return false;
//            }

        } catch (AmazonServiceException e) {
            throw e;
        } finally {
            if (sourceS3Object != null) {
                sourceS3Object.close();
            }
        }
        //System.out.println("Copied successfully to " + destS3Uri.toString() + " Etag:" + eTag);
        return true;
    }

But I am getting the below error:

{ "errorMessage": "Data read has a different length than the expected: dataLength=964481363; expectedLength=964481376; includeSkipped=false; in.getClass()=class com.amazonaws.internal.ReleasableInputStream; markedSupported=false; marked=0; resetSinceLastMarked=false; markCount=0; resetCount=0", "errorType": "com.amazonaws.SdkClientException", "stackTrace": [ "com.amazonaws.util.LengthCheckInputStream.checkLength(LengthCheckInputStream.java:151)", "com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:109)", "java.io.FilterInputStream.read(FilterInputStream.java:107)", "com.amazonaws.services.s3.internal.crypto.CipherLiteInputStream.nextChunk(CipherLiteInputStream.java:225)", "com.amazonaws.services.s3.internal.crypto.CipherLiteInputStream.read(CipherLiteInputStream.java:118)", "com.amazonaws.services.s3.internal.crypto.RenewableCipherLiteInputStream.read(RenewableCipherLiteInputStream.java:112)", "com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)", "com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)", "com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107)", "com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)", "com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.read(MD5DigestCalculatingInputStream.java:128)", "java.io.BufferedInputStream.fill(BufferedInputStream.java:246)", "java.io.BufferedInputStream.read1(BufferedInputStream.java:286)", "java.io.BufferedInputStream.read(BufferedInputStream.java:345)", "com.amazonaws.internal.SdkBufferedInputStream.read(SdkBufferedInputStream.java:76)", "com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)", "com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)", "com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)", "org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)", "com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:160)", "org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)", "org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:162)", "org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)", "com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:63)", "org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)", "org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)", "org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)", "org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)", "org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)", "org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)", "com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)", "com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1297)", "com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)", "com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)", "com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)", "com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)", "com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)", "com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)", "com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)", "com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)", "com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4914)", "com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4860)", "com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:389)", "com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5793)", "com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1786)", "com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1746)", "com.amazonaws.services.s3.AmazonS3EncryptionClient.access$101(AmazonS3EncryptionClient.java:81)", "com.amazonaws.services.s3.AmazonS3EncryptionClient$S3DirectImpl.putObject(AmazonS3EncryptionClient.java:690)", "com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.putObjectUsingMetadata(S3CryptoModuleBase.java:175)", "com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.putObjectSecurely(S3CryptoModuleBase.java:161)", "com.amazonaws.services.s3.internal.crypto.CryptoModuleDispatcher.putObjectSecurely(CryptoModuleDispatcher.java:108)", "com.amazonaws.services.s3.AmazonS3EncryptionClient.putObject(AmazonS3EncryptionClient.java:570)", "com.amazon.testCase.lambda.object.UploadObject.coyS3Object(UploadObject.java:93)", "com.amazon.testCase.lambda.object.UploadObject.handleRequest(UploadObject.java:54)", "sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)", "sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(Method.java:498)" ] }

When you said you read the source length, you are reading encrypted object length right?. And the header you are showing is actually unencrypted content length. They will definitely be different — karthick
possible duplicate of stackoverflow.com/questions/27798825/… — Scary Wombat

constantLeaner constantLeaner · Accepted Answer · 2019-03-29T05:12:48

I figured out the problem. The content length of S3 object is the encrypted data length. But when I am putting the object, I am trying to put the unencrypted data. So I need to explicitly set the content length of the object to the unencrypted length. Below code solves the issue.

 try {
            //Get the Source object stream
            sourceS3Object = sourceS3Client.getObject(sourceS3Uri.getBucket(), sourceS3Uri.getKey());
            //For a decrypted object, the content length in metadata has the encrypted length
            //Set the content length to the unencrypted-data-length
            ObjectMetadata objectMetadata = sourceS3Object.getObjectMetadata();
            Map<String, String> modelMetaData = objectMetadata.getUserMetadata();
            Long unencryptedDataLength = Long.parseLong(modelMetaData.get(
                    "x-amz-unencrypted-content-length"));
            objectMetadata.setContentLength(unencryptedDataLength);
            PutObjectRequest putRequest = new PutObjectRequest(destS3Uri.getBucket(), destS3Uri.getKey(),
                    sourceS3Object.getObjectContent(), objectMetadata);

            eTag = destS3Client.putObject(putRequest).getETag();

            if (eTag.isEmpty()) {
                System.out.println("Copy failed, New object in " + destS3Uri.toString() + " is empty");
                return false;
            }

Data read has a different length than the expected: dataLength=964481363; expectedLength=964481376;

1 Answers