Is "in the cloud" gsutil cp an atomic operation?

Question

Assuming I have copied one object into a Google Cloud Storage bucket using the following command:

gsutil -h "Cache-Control:public,max-age=3600" cp -a public-read a.html gs://some-bucket/

I now want to copy this file "in the cloud" while keeping the public ACL and simultaneously updating the Cache-Control header:

gsutil -h "Cache-Control:no-store" cp -p gs://some-bucket/a.html gs://some-bucket/b.html

Is this operation atomic? I.e. can I be sure, that the object gs://some-bucket/b.html will become initially available with the modified Cache-Control:no-store header?

The reason for my question is: I'm using a Google Cloud Storage bucket as a CDN-backend. While I want most of the objects in the bucket to be cached by the CDN according to the max-age provided in the Cache-Control header I want to make sure that a few specific files, which are in fact copies of cacheable versions, are never cached by a CDN. It is therefore crucial that these objects – when being copied – never appear with a Cache-Control:public,max-age=XXX but immediately appear with a Cache-Control:no-store header as to eliminate the chance that a request coming from a CDN would read the copied object at a point in time where a max-age would still be present and hence cache the object which is supposed to never be cached.

jterrace jterrace · Accepted Answer · 2017-03-02T15:49:32

Yes, copying to the new object with Cache-Control set will be atomic. You can verify this by looking at the metageneration property of the object.

For example, upload an object:

$ BUCKET=mybucket
$ echo foo | ./gsutil cp - gs://$BUCKET/foo.txt
Copying from <STDIN>...
/ [1 files][    0.0 B/    0.0 B]                                                
Operation completed over 1 objects.

and you'll see that its initial metageneration is 1:

$ ./gsutil ls -L gs://$BUCKET/foo.txt | grep Meta
    Metageneration:         1

Whenever an object's metadata is modified, the metageneration is changed. For example, if the cache control is updated later like so:

$ ./gsutil setmeta -h "Cache-Control:no-store" gs://$BUCKET/foo.txt
Setting metadata on gs://mybucket/foo.txt...
/ [1 objects]                                                                   
Operation completed over 1 objects.

The new metageneration is 2:

$ ./gsutil ls -L gs://$BUCKET/foo.txt | grep Meta
    Metageneration:         2

Now, if we run the copy command:

$ ./gsutil -h "Cache-Control:no-store" cp -p gs://$BUCKET/foo.txt gs://$BUCKET/bar.txt
Copying gs://mybucket/foo.txt [Content-Type=application/octet-stream]...
- [1 files][    4.0 B/    4.0 B]                                                
Operation completed over 1 objects/4.0 B.

The metageneration of the new object is 1:

$ ./gsutil ls -L gs://$BUCKET/bar.txt | grep Meta
    Metageneration:         1

This means that the object was written once and has not been modified since.

Is "in the cloud" gsutil cp an atomic operation?

1 Answers