ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden when trying cross account copy

0
votes

I am doing a cross account copy of s3 objects. When I am trying to copy files from source bucket to destination bucket I am getting the error ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden I am getting error at line s3_object.download_fileobj(buffer)

    final_df=pd.DataFrame() 
for file in files1: 
# file=file.split('/')[-1]
    bucket = 'source bucket'
    buffer = io.BytesIO()
    s3 = boto3.resource('s3')
    s3_object = s3.Object(bucket,file)
    s3_object.download_fileobj(buffer)
    df = pd.read_parquet(buffer)
    print(file)
    s3 = boto3.client('s3')
    file=file.split('/')[-1]
    print(file)
    final_df=pd.concat([final_df,df],sort=False)

Files1 is the list of all parquet files in the bucket

1
pythonamazon-web-servicesamazon-s3boto3amazon-sagemaker

1 Answers

1
votes

The issue here may be with this being a cross account copy. You may need to set up an IAM role to ensure that both accounts have permission.

As found in https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/, along with other troubleshooting options,

For ongoing cross-account permissions, create an IAM role in your account with permissions to your bucket. Then, grant another AWS account the permission to assume that IAM role. For more information, see Tutorial: Delegate access across AWS accounts using IAM roles. (https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html)