Migrate active storage from local disk service to gcs cloud

Question

I'm trying to migrate my local Active Storage files to Google Cloud Storage. I tried to just copy the files of /storage/* to my GCS Bucket - but it seems that this does not work.

I get 404 not found errors cause it is searching for files like: [bucket]/variants/ptGtmNWuTE...

My local storage directory has a totally different folder structure with folders like: /storage/1R/3o/NWuT....

My method to retrieve the image is as followed:

variant = attachment.variant(resize: '100x100').processed
url_for(variant)

What am i missing here?

Oliver Oliver · Accepted Answer · 2019-03-13T12:17:01

As it turns out - DiskService aka. local storage uses a different folder structure than the cloud services. Thats really weird.

DiskService uses as folders part of the first chars of the key. Cloud services just use the key and put all variants in a separate folder.

Created a rake task to copy files over to cloud services. Run it with rails active_storage:migrate_local_to_cloud storage_config=google for example.

namespace :active_storage do
  desc "Migrates active storage local files to cloud"
    task migrate_local_to_cloud: :environment do
      raise 'Missing storage_config param' if !ENV.has_key?('storage_config')

      require 'yaml'
      require 'erb'
      require 'google/cloud/storage'

      config_file = Pathname.new(Rails.root.join('config/storage.yml'))
      configs = YAML.load(ERB.new(config_file.read).result) || {}
      config = configs[ENV['storage_config']]

      client = Google::Cloud.storage(config['project'], config['credentials'])
      bucket = client.bucket(config.fetch('bucket'))

      ActiveStorage::Blob.find_each do |blob|
        key = blob.key
        folder = [key[0..1], key[2..3]].join('/')
        file_path = Rails.root.join('storage', folder.to_s, key)
        file = File.open(file_path, 'rb')
        md5 = Digest::MD5.base64digest(file.read)
        bucket.create_file(file, key, content_type: blob.content_type, md5: md5)
        file.close
        puts key
      end
    end
  end

Migrate active storage from local disk service to gcs cloud

1 Answers