"gsutil -m mv" not running parallel transfers

Question

I am moving a large number of files from a local drive into a bucket using "gsutil -m mv". However during the transfer process it would appear to only be running one transfer at a time. I have checked top and only see one process from python running the command. I have modified both "parallel_process_count" "parallel_thread_count" in the boto config file and do not observe any change in the transfers behavior. Even when running gsutil with the -m option i still receive the message below:

"==> NOTE: You are performing a sequence of gsutil operations that may run significantly faster if you instead use gsutil -m -m ... Please see the -m section under "gsutil help options" for further information about when gsutil -m can be advantageous."

Has anyone else run into this issue before?

OS: Centos 6.6 gsutil version: 4.15 python version: 2.6.6

What are your system specs in terms of CPU / RAM, and what kind of load of files are you trying to upload? — Nick
The system itself is a VM configured with 8 cores and 24GB of ram. This can easily be increased if needed. However, as gsutil doesn't seem to be multi threading its just barely scratching the surface in terms of resource utilization. As for the files themselves. Its roughly 15TB consisting of many files ranging from <5mb to 10s of GB. — oishiimsg

Travis Hobrla Travis Hobrla · Accepted Answer · 2015-10-01T17:27:17

This is a bug in gsutil 4.14-4.15, where the -m flag was not getting propagated correctly for the mv command. It is fixed in gsutil 4.16.

"gsutil -m mv" not running parallel transfers

1 Answers