Streaming download on S3 from an external private bucket to company bucket with rails

Question

Our company is using Ruby 2.1.3 with AWS SDK V1 for uploading files on S3. I need to stream files directly from a private external bucket to one of our personal bucket (without actually downloading them locally). I can't find any good documentation on the subject.

The copy_from method provided by the SDK, I think, does not permit streaming from a private external bucket to one of our bucket.

We have tried using open-uri to stream the download and stream the upload to s3 but the file was always downloaded fully first and then uploaded (is it supposed to be like that?).

Any help is welcomed!

Thank you.

Answer 1

The V1 SDK doesn't allow you to transfer between buckets directly. You can do what open-uri does and download the file and then upload to the new bucket.

If you want a solution that can still work in Ruby I suggest using the AWS CLI. You can add a line like this to your code:

`aws s3 cp s3://frombucket/ s3://tobucket/`

The backticks allow you to execute system commands in your ruby script. Alternatively you could upgrade to the V2 SDK and use the object.copy_to command and copy between buckets. Hope this helps!