Move/copy data from one folder to another on AWS S3

Question

I am looking for all the methods for moving/copying the data from one folder to another on AWS S3 bucket.

Method 1: Via AWS CLI (Most easy)

Download and install awscli on ur instance, I am using here windows(64-bit link) and run "asw configure" to fill up your configuration and just run this single command on cmd

aws s3 cp s3://from-source/ s3://to-destination/ --recursive

Here cp for copy and recursive to copy all files

Method 2: Via AWS CLI using python

import os
import awscli
if os.environ.get('LC_CTYPE', '') == 'UTF-8':
    os.environ['LC_CTYPE'] = 'en_US.UTF-8'

from awscli.clidriver import create_clidriver
driver = create_clidriver()
driver.main('s3 mv s3://staging/AwsTesting/research/    s3://staging/AwsTesting/research_archive/ --recursive'.split())

Even this worked for me perfectly

Method 3: Via Boto using python

import boto3
s3 = boto3.resource('s3')
copy_source = {
    'Bucket': 's3://staging/',
    'Key': 'AwsTesting/research/'
}
s3.meta.client.copy(copy_source, ''s3://staging/'', 'AwsTesting/research_archive/')

With my understanding I have assumed the 'key' for bucket is just the folder prefix so I have mentioned the folder path here

Error:

Invalid bucket name "s3://staging": Bucket name must match the regex "^[a-zA-Z0-9.-_]{1,255}$"

Even I changed it to simple bucket name as "staging" but no success.

How can I understand bucket connectivity via boto and the concept of this key?

I see no reason in the docs to believe that s3.meta.client.copy copies anything other than a single object. "Copy an object from one S3 location to another. This is a managed transfer which will perform a multipart copy in multiple threads if necessary." A "multipart copy" does not mean multiple objects. It means that a single object is copied using multiple HTTP transactions, each handling different byte ranges. — Michael - sqlbot
s3 bucket names have to be globally unique, so you cannot have a bucket of the same name as someone else... by now if noone already owns the single s3 bucket called 'staging' I'd be very surprised! — danimal

raisashi raisashi · Accepted Answer · 2018-01-16T17:42:26

import boto3
s3 = boto3.resource('s3')
copy_source = {
    'Bucket': 'staging',
    'Key': 'AwsTesting/research/filename.csv'
}
s3.meta.client.copy(copy_source, 'staging', 'AwsTesting/')

Move/copy data from one folder to another on AWS S3

2 Answers