Can I delete container images from Google Cloud Storage artifacts bucket?

40
votes

I have a Google App Engine app, which connects to Google Cloud Storage.

I noticed that the amount of data stored was unreasonably high (4.01 GB, when it should be 100MB or so).

So, I looked at how much each bucket was storing, and I found that there was an automatically created bucket called us.artificats. that was taking up most of the space.

I looked inside, and all it has is one folder: containers/images/.

From what I've Googled, it seems like these images come from Google Cloud Build.

My question is, can I delete them without compromising my entire application?

6
google-app-enginegoogle-cloud-platformgoogle-cloud-storagegoogle-cloud-build
Only delete container images using the Console or the CLI gcloud container images delete. A container consists of layers. You are seeing these layers as objects in Cloud Storage. Do not directly delete these objects. In simple terms, these objects are cached layers that are used to build a container image. For some tools if a layer is not cached it will first be pulled (downloaded), but I have not tried to deliberately delete container registry objects to see what happens. – John Hanley
Thank you for the clarification! What do I use as my image name? – Caleb H.
gcloud container images list says there are no images... – Caleb H.
Do you have more than one repository? Use the --repository=[HOSTNAME]/[PROJECT-ID] Repository locations are gcr.io, us.gcr.io, eu.gcr.io, and asia.gcr.io. Also review this document: cloud.google.com/container-registry/docs/managing – John Hanley
@EvanBaldonado I got that from the number and size of the files I was intentionally uploading. – Caleb H.

6 Answers

40
votes

I have solved this problem by applying a deletion rule. Here's how to do it:

  1. Open the project in Google Cloud console
  2. Open the storage management (search for "Storage" for example).
  3. In the Browser tab, select the container us.artifacts....
  4. Now, open the Lifecycle section. You should see something like:

enter image description here

  1. Click on Add a rule and provide the following conditions:
    1. In the action, select Delete object
    2. In the conditions, select Age and enter for example 3 days
  2. Click on create to confirm the creation

Now all objects older than 3 days will be automatically deleted. It might take a few minutes for this new rule to be applied by Google Cloud.

20
votes

For those of you seeing this later on, I ended up deleting the folder, and everything was fine.

When I ran Google Cloud Build again, it added items back into the bucket, which I had to delete later on.

As @HarshitG mentioned, this can be set up to happen automatically via deletion rules in cloud storage. As for myself, I added a deletion step to my deployment GitHub action.

12
votes

Here is the reference to the documentation. https://cloud.google.com/appengine/docs/standard/go/testing-and-deploying-your-app#managing_build_images

Built container images are stored in the app-engine folder in Container Registry. You can download these images to keep or run elsewhere. Once deployment is complete, App Engine no longer needs the container images. Note that they are not automatically deleted, so to avoid reaching your storage quota, you can safely delete any images you don't need. For more information about managing images in Container Registry, see the Container Registry documentation.

This can be automated by adding a Lifecycle rules like @HarshitG mentioned. enter image description here

5
votes

Same issue. Thanks for the update, Caleb.

I'm having the same issue, but I don't have an App running; I just have:

  • Firebase Auth
  • Firestore
  • Firebase Functions
  • Cloud Storage

Not sure why I have 4GB stored in those containers, and I'm not sure if I should delete them or if that would break my functions.

UPDATE: I deleted the container folder and all still works. Not sure if those are backups or whatnot, but I can't find anything online or in the docs. I will post here if something happens. As soon as a cloud function ran, the folder had 33 files again.

5
votes

You can add a trigger to your lifecycle rules on console.cloud.google.com.

  1. Access the bucket with artifacts (default is "us.artifacts.yourAppName.appspot.com")
  2. Go to "Life cycle".
  3. Click on "Add a rule".
  4. Check "Object delete" and press "Continue".
  5. Check the filter to delete the bucket, I chose "age" and selected three days as the number of auto delete old elements (after element has 3 days of life it's auto deleted).
  6. Click on "Create" and the rule is working now, then you do not need to visit every day to clean the bucket.
0
votes

I recommend against setting up a lifecycle rules on your Storage buckets. It will likely lead to breaking subsequent updates to the function (as described in Cloud Function build error - failed to get OS from config file for image)

If you are interested in cleaning up container images, you should instead delete container images stored in Google Cloud Registry https://console.cloud.google.com/gcr. Deleting container images on GCR repos will automatically cleanup objects stored in your Cloud Storage.

https://issuetracker.google.com/issues/186832976 has relevant information from a Google Cloud Functions engineer.