2
votes

I am using GNU Parallel to download 1000 files from S3 to two EC2 hosts (-S:,R), where 'R' is IP of remote peer.
With --jobs 64, it takes a long time to download 64 files (rather large files) from S3:

time parallel --jobs 64 --joblog out.txt -S:,R --env HTTPS_PROXY --env S3_DIR "aws s3 cp ${S3_DIR}part-{}.gz /my/path/" ::: `seq -f "%05g" 0 1000`

As the result, I get

Connection to R closed by remote host.

This is due to sshd configuration on the remote peer, and I cannot change these settings (machines are not mine.)

Looking at the documentation, I cannot find anything like keep-alive. How can I work around this issue? Thank you.

1

1 Answers

0
votes

You may not be able to change sshd_config, but often it will respect options given by ssh:

ssh -o TCPKeepAlive ...

If this is the case here, you can make GNU Parallel use a different ssh command:

parallel -S '/path/to/different/ssh_command -o KeepAlive server' echo ::: foo