Access Denied while putting file from HDFS to S3 bucket

Question

I have written a Java program which is included in the oozie workflow which puts files from HDFS to S3 bucket. However, I am getting the following error

com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 310F08CD4FF8B5D9), S3 Extended Request ID: fAysD1vgtriV8x+sf1zqHk58eAT89Y6HD+ziEokaPvFPKwaPrHDxt5yygsiA1ktNVsyj+GTmbQ0=

I am creating the key path in S3 bucket dynamically in oozie workflow.
For eg: If my file name is abc_20171009.tsv.gz then this file should be uploaded to bucket in the following path

tsvFile/year=2017/month=10/day=09/abc_20171009.tsv.gz

In the similar way other day files should be uploaded based on the date.
My query is whether the key path should preexist in the bucket before uploading the files or it can be created dynamically?

 // Request server-side encryption.           

          BasicAWSCredentials awsCredentials = new BasicAWSCredentials(awsAccessKeyId, awsSecretKey);
          AmazonS3Client s3Client = new AmazonS3Client(awsCredentials);
          PutObjectRequest request  = new PutObjectRequest("bucket_name", "key_name","");

          ObjectMetadata objectMetadata  = new ObjectMetadata();
          objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);

          request.setMetadata(objectMetadata); 

          PutObjectResult response = s3Client.putObject(request);

          LOGGER.info("Server Side Encryption successful" +response.getSSEAlgorithm());

Note:I am able to manually put the file and connect to S3 bucket through AWS CLI.

pezetem pezetem · Accepted Answer · 2017-10-09T06:52:50

the amazon S3 serive have a list of safe characters which might be used for S3 object key

Safe Characters The following character sets are generally safe for use in key names: • Alphanumeric characters [0-9a-zA-Z] • Special characters !, -, _, ., *, ', (, and )

however what I can see, you are using =, which on the other hand is a special character requiring handling

Characters That Might Require Special Handling The following characters in a key name may require additional code handling and will likely need to be URL encoded or referenced as HEX. Some of these are non-printable characters and your browser may not handle them, which will also require special handling

and one of those characters is =

perhaps, when you upload file manually, the = is automatically encoded. So you have to options, remove thos = signs or encode them (I would go with the first options)

Access Denied while putting file from HDFS to S3 bucket

1 Answers