Rsync with AWS lambda(java) between Bucket and Remote Server

Question

I want to perform an one direction rsync between an AWS S3 Bucket and a remote ftp server (accepts ftps) with a java lambda function. So if one file in bucket is deleted the lambda cron should remove it from the remote ftp server. I read that aws cli offers the function s3 sync. Could this be an option?

best regards Jannik

Which method should I use of the TransferManager? the copy function? — JavaTourist

stdunbar stdunbar · Accepted Answer · 2019-12-05T22:04:29

This would be pretty straight forward. The Lambda would be setup to be triggered on an S3 delete. The basic code (untested) would be something like:

public class Handler implements RequestHandler<S3Event, String> {

    public String handleRequest(S3Event s3event, Context context) {
        try {
            S3EventNotificationRecord record = s3event.getRecords().get(0);

            // Object key may have spaces or unicode non-ASCII characters.
            String srcKey = record.getS3().getObject().getUrlDecodedKey();

            // now use Apache Commons Net
            // (https://commons.apache.org/proper/commons-net/)
            // to delete the file on the FTP server

            FTPClient ftpClient = new FTPClient();

            ftpClient.connect(server, port);

            int replyCode = ftpClient.getReplyCode();
            if (!FTPReply.isPositiveCompletion(replyCode)) {
                contect.getLogger().log("SFTP Connect failed");
                return;
            }

            boolean success = ftpClient.login(user, pass);

            if (!success) {
                contect.getLogger().log("Could not login to the FTP server");
                return;
            }
            String fileToDelete = "/some/ftp/directory/" + srcKey;

            boolean deleted = ftpClient.deleteFile(fileToDelete);
            if (deleted) {
                contect.getLogger().log("The file was deleted successfully.");
            } else {
                contect.getLogger().log("Could not delete the  file, it may not exist.");
            }
        }
        catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

On the S3 side, you will need to enable your S3 bucket to send a delete event to your Lambda. This can be done in the AWS console by selecting the bucket and in the advanced section, add select Events, add a notification, select "Permanently deleted" (or "All object delete events") and add your Lambda.

Rsync with AWS lambda(java) between Bucket and Remote Server

1 Answers