Saving pandas dataframe variable to csv from google compute engine to google storage bucket without saving to disk first

Question

I have a google cloud storage bucket called google-storage-bucket-1.

I'm connected my compute engine instance and I have a pandas dataframe variable which is created in python as a temporary variable called df1.

I want to save the dataframe as a csv file into the bucket. I use the following command.

import pandas as pd

df1.to_csv('gs://google-storge-bucket-1/test/dataframe1.csv')

But I get the following error,

OSError: Forbidden: https://www.googleapis.com/upload/storage/v1/b/xxx/o
Insufficient Permission

Whats the proper command to save the file to the bucket without saving it to disk first?

Answer 1

Like this:

from google.cloud import storage
import os
from io import StringIO

f = StringIO()  ## this is to avoid creating local file
df1.to_csv(f)
f.seek(0)

gcs = storage.Client()
gcs.get_bucket('google-storge-bucket-1').blob('dataframe1.csv').upload_from_file(f, content_type='text/csv')