Loading multiple files from Google Cloud Storage into a single Pandas Dataframe

Question

I have been trying to write a function that loads multiple files from a Google Cloud Storage bucket into a single Pandas Dataframe, however I cannot seem to make it work.

import pandas as pd
from google.datalab import storage
from io import BytesIO


def gcs_loader(bucket_name, prefix):
  bucket = storage.Bucket(bucket_name)
  df = pd.DataFrame()
  for shard in bucket.objects(prefix=prefix):
    fp = shard.uri
    %gcs read -o $fp -v tmp
    df.append(read_csv(BytesIO(tmp))
  return df

When I try to run it says:

undefined variable referenced in command line: $fp

I'm having the same problem. The interesting thing is that when the code is not run inside a function, the magic works fine. My conclusion is that it must be a restriction of placing the magic commands inside functions, but I haven't been able to find information about it. — jimijazz

Bob Smith Bob Smith · Accepted Answer · 2017-10-27T16:11:36

Sure, here's an example: https://colab.research.google.com/notebook#fileId=0B7I8C_4vGdF6Ynl1X25iTHE4MGc

This notebook shows the following:

Create two random CSVs
Upload both CSV files to a GCS bucket
Uses the GCS Python API to iterate over files in the bucket. And,
Merge each file into a single Pandas DataFrame.

Loading multiple files from Google Cloud Storage into a single Pandas Dataframe

1 Answers