creating a file in google cloud storage - IOError: Buffer is closed

Question

I created a python script to pull the data from facebookads API and create a file in google cloud storage by using google app engine.

getting the following error while writing the data to google cloud storage but data is displaying properly on web browser:

IOError: Buffer is closed.

After some research I understood that, this error will come when not able to recognize end of the lin ("\n") , so it treats the entire file as a single line and raise "Buffer is Closed" error.

So I added following code and now displaying rows properly on web browser but still getting error while writing into gcs.

data1=data.replace("\n", "<br />")

Code:

class get_region_insights(webapp.RequestHandler):

    _apptitle      = None
    _projectid     = None
    _projectnumber = None


    def get(self):
        #bucket_name = os.environ.get('BUCKET_NAME', app_identity.get_default_gcs_bucket_name())
        cfg=appsettings()
        for i in cfg._templates:

            id=int(i['_id'])

            if id == 7:

                ### Read variables from config file 
                bucket_name = i['_gcsbucket']
                bucket = '/' + bucket_name
                filename = bucket + '/' + i['_filename'] + str(time.strftime("%d_%m_%Y")) + ".csv"
                ad_acct=i['_ad_acct']
                app_id = i['_app_id']
                app_secret = i['_app_secret']
                access_token = i['_access_token']
                needed_keys=ast.literal_eval(i['_needed_keys'])
                self.tmp_filenames_to_clean_up = []

                u = date.today()
                sub_days = 1
                s = u - timedelta(sub_days)
                until = str(u)
                since = str(s)

                params = {
                    'fields': [
                        FBAdsInsights.Field.account_id,
                        FBAdsInsights.Field.campaign_id,
                        FBAdsInsights.Field.campaign_name,
                        FBAdsInsights.Field.adset_id,
                        FBAdsInsights.Field.adset_name,
                        FBAdsInsights.Field.impressions,
                        FBAdsInsights.Field.spend,
                        FBAdsInsights.Field.actions,
                        FBAdsInsights.Field.action_values,
                    ],
                    'filtering': [
                        {
                            'field': 'action_type',
                            'operator': 'IN',
                            'value': ["link_click","comment", "post_reaction", "post", "offsite_conversion.fb_pixel_purchase"] #custom rule filter
                            },        
                        ],    
                    'level': 'adset',
                    'time_range': {
                         'since': since, #user input field
                         'until': until #specify dynamic date range between (today() - (days_entered)) and today()
                        },
                    'time_increment': 1,
                    'breakdowns': ['region'],
                    'action_breakdowns': ['action_type'],
                }

                ### Initializing Google cloud Storage Object 
                write_retry_params = _gcs.RetryParams(backoff_factor=1.1)
                gcs_file=_gcs.open(filename, 'w', content_type='text/plain',retry_params=write_retry_params)


                ### Facebook Initialization 
                session=FacebookSession(app_id,app_secret,access_token)
                api=FacebookAdsApi(session)
                FacebookAdsApi.set_default_api(api)
                ad_account = FBAdAccount(ad_acct)
                stats = ad_account.get_insights(params=params,async=True)
                stats.remote_read()
                while stats[AdReportRun.Field.async_percent_completion] < 100:
                    time.sleep(1)
                    stats.remote_read()
                    time.sleep(1)
                stats1 = stats.get_result()
                x = [x for x in stats1]

                ### Printing the result and writing to Google Cloud Storage

                for i in x:
                    for k in i['actions']:
                        if k['action_type'] == "offsite_conversion.fb_pixel_purchase":
                            Purchase_Facebook_Pixel = k['value']
                        if k['action_type'] == "comment":
                            post_comment= k['value']
                        if k['action_type'] == "link_click":
                            link_click=k['value']
                        if k['action_type'] == "post":
                            post_share=k['value']
                        if k['action_type'] == "post_reaction":
                            post_reaction=k['value']
                    for m in i['action_values']:
                        if m['action_type'] == "offsite_conversion.fb_pixel_purchase" :
                            Purchase_Conversion_Value_Facebook_Pixel=m['value']
                    data=(i['account_id'] + "," + i['adset_id'] + "," + i['campaign_id'] + "," + i['date_start'] + "," + i['date_stop'] + ","+ i['impressions']+ "," + i['region'] + ","+ i['spend']+ "," + link_click + "," + Purchase_Facebook_Pixel + "," + Purchase_Conversion_Value_Facebook_Pixel+"\n")
                    data1=data.replace("\n", "<br />")
                    self.response.write(data.replace("\n", "<br />"))
                    #self.response.write("\n"+i['account_id'] + "," + i['adset_id'] + "," + i['adset_name'] + "," + i['age'] + "," + i['campaign_id'] + "," +i['campaign_name'] + "," + i['date_start'] + "," + i['date_stop'] + ","+i['gender'] + ","+ i['impressions']+","+ i['spend']+ "," + link_click + "," + post_comment + "," + post_share + "," + post_reaction + "," + Purchase_Facebook_Pixel + "," + Purchase_Conversion_Value_Facebook_Pixel+"\n")
                    gcs_file.write(data1.encode('utf-8'))
                    gcs_file.close()
                    self.tmp_filenames_to_clean_up.append(filename)

snakecharmerb snakecharmerb · Accepted Answer · 2017-06-15T05:33:22

You are opening the cloud storage file outside your loop, but then you close inside the loop.

            ### Initializing Google cloud Storage Object 
            write_retry_params = _gcs.RetryParams(backoff_factor=1.1)
            gcs_file=_gcs.open(filename, 'w', content_type='text/plain',retry_params=write_retry_params)


            ### Facebook Initialization 
            ...

            ### Printing the result and writing to Google Cloud Storage

            for i in x:
                # do stuff with data
                ...
                gcs_file.write(data1.encode('utf-8'))
                gcs_file.close()   # <-- closing the file buffer
                self.tmp_filenames_to_clean_up.append(filename)

If you want to write one file for each loop iteration, open and close the file inside the loop.

If you want to to write all the data to a single file, open and close the file outside the loop.

creating a file in google cloud storage - IOError: Buffer is closed

1 Answers