Google cloud storage lifecycle configuration filesize?

Question

I've been trying to implement lifecycle management to objects I store in my GCP storage with no overall success.

First of all I have a bucket that stores objects per date. (intermediate & files of pipeline runs) So I want to retain, say maybe 10 days past, but then I want to move to more archived uses.

So on this storage bucket I've created the following lifecycle.json:

{
   "lifecycle":{
      "rule":[
         {
            "action":{
               "type":"SetStorageClass",
               "storageClass":"NEARLINE"
            },
            "condition":{
               "age":3,
               "matchesStorageClass":[
                   "REGIONAL",                  
                  "STANDARD",
                  "DURABLE_REDUCED_AVAILABILITY"
               ]
            }
         },
         {
            "action":{
               "type":"SetStorageClass",
               "storageClass":"COLDLINE"
            },
            "condition":{
               "age":10,
               "matchesStorageClass":[
                  "NEARLINE"
               ]
            }
         },
         {
            "action":{
               "type":"Delete"
            },
            "condition":{
               "age":10,
               "matchesStorageClass":[
                  "COLDLINE"
               ]
            }
         }
      ]
   }
}

Based on this manual: https://cloud.google.com/storage/docs/managing-lifecycles#enable

When I set the lifecycle using gsutil it accepts it, and when I get it it shows it.

But not all files seem to be affected by the configuration and I'm having this theory that it depends on filesize, e.g.

All files were uploaded on the same exact day (with 1 hour window). but only some are nearline, some are still regional. In this directory the tipping point is around 100MB, but in another a 300MB+ file makes it into nearline

My questions:

How do I make this consistent?
Are some of the files still being processed perhaps?
Can I monitor this process anywhere?
If skipped now, how do I retrigger for those files, because I still have files in this bucket that are +1GB in size, but more than 23 days old and still present in the regional bucket

--- Update 20170720 ---

As per requested I've ran the command:

gsutil ls -L gs://yourbucket

One large csv file, which should be removed by now:

Creation time:          Tue, 27 Jun 2017 12:23:08 GMT
Update time:            Tue, 27 Jun 2017 12:23:08 GMT
Storage class:          REGIONAL
Content-Length:         1184976409
Content-Type:           text/csv
Component-Count:        23
Hash (crc32c):          ggHfjw==
ETag:                   COqC4oqC3tQCEAE=
Generation:             1498566188630378
Metageneration:         1

The directory of files from the screenshot has been processed and everything was deleted below a certain filesize:

The files remaining should also have been deleted:

Creation time:          Sat, 08 Jul 2017 10:00:57 GMT
Update time:            Sat, 08 Jul 2017 10:00:57 GMT
Storage class:          REGIONAL
Content-Length:         469342393
Content-Type:           application/octet-stream
Component-Count:        9
Hash (crc32c):          Ux3HKw==
ETag:                   COLb3ei2+dQCEAE=
Generation:             1499508057271778
Metageneration:         1
ACL:                    [

So we can rule out the fact that the file was modified in the mean time.

Note that the browser screenshot you're showing is displaying "Last modified", which comes from the updated timestamp field. Lifecycle uses the timeCreated timestamp field for the Age condition. Can you run gsutil ls -L gs://yourbucket and confirm that timeCreated is what you expect? — jterrace

Tom Lous Tom Lous · Accepted Answer · 2018-04-03T12:04:53

0

votes

This seems to have been fixed in the mean time

Google cloud storage lifecycle configuration filesize?

1 Answers