3
votes

How we can create the folder using gsutil command. I am using Bashoperator in airflow where I need to use the gsutil Bash command, Bucket is already created I want to create a folder inside bucket. I already tried with below command but It's not working for me.

$ gsutil cp <new_folder> gs://<bucketname>/  

I am getting error - CommandException: No URLs matched: new_folder

2

2 Answers

2
votes

Google Storage does not work like a regular file system as in Windows/Linux. It appears to have folders but in the background it behaves as it does not. It only allows us to create "folders" so we can organize better and for our comfort.

If you want to save data in specific folders from gsutil try this.

gsutil cp [filetocopy] gs://your-bucket/folderyouwant/your-file

It will store the item in a "folder".

Check this link for more gsutil cp information.

This is the logic behind Google Cloud Storage "Folders".

gsutil will make a bucket listing request for the named bucket, using delimiter="/" and prefix="abc". It will then examine the bucket listing results and determine whether there are objects in the bucket whose path starts with gs://your-bucket/abc/, to determine whether to treat the target as an object name or a directory name. In turn this impacts the name of the object you create: If the above check indicates there is an "abc" directory you will end up with the object gs://your-bucket/abc/your-file; otherwise you will end up with the object gs://your-bucket/abc.

Here you have more interesting information about this if you want.

0
votes

Apparently the ability to create an empty folder using gsutil is a request that has been seen a few times but not yet satisfied. There appears to be some workarounds by using API that can then be scripted. The GitHub issue for the ability to create empty folders through scripting can be found here:

https://github.com/GoogleCloudPlatform/gsutil/issues/388