Node.js Cloud Function - Stream CSV data directly to Google Cloud Storage file

1
votes

I have a script that can call a RESTful API and retrieve CSV data from a report in chunks. I'm able to concatenate, parse, and display this data in the console. I am also able to write this CSV data to a local file and store it.

What I am trying to figure out is how to skip creating a file to store this data before uploading it to GCS and instead transfer it directly into Google Cloud Storage to save as a file. Since I am trying to make this a serverless cloud function, I am trying to stream it directly from memory into a Google Cloud Storage file.

I found this 'Streaming Transfers' documentation on google, but it only references doing this with 'gsutil' and I am struggling to find any examples or documentation on how to do this with node.js. I also tried to follow this answer on Stack overflow, but it's from 2013 and the methods seem a little out-dated. My script also isn't user-facing, so I don't need to hit any routes.

I am able to upload local files directly to my bucket using the function below, so Authentication isn't an issue. I'm just unsure how to convert a CSV blob or object in memory into a file in GCS. I haven't been able to find many examples so wasn't sure if anyone else has solved this issue in the past.

const { Storage } = require('@google-cloud/storage');
const storage = new Storage({
  projectId,
  keyFilename
 });

function uploadCSVToGCS() {
   const localFilePath = './test.csv';
   const bucketName = "Test_Bucket";
   const bucket = storage.bucket(bucketName);

   bucket.upload(localFilePath);
};

I also found a 3rd party plugin that Google references called 'boto' that seems to do what I want, but this is for python, not node.js unfortunately.

2
node.jsuploadgoogle-cloud-functionsgoogle-cloud-storage

2 Answers

2
votes

Streaming object data to Cloud Storage is illustrated in the documentation. You will need to understand how node streams work, and make use of createWriteStream. The sample code is not exactly what you want, but you'll use the same pattern:

function sendUploadToGCS (req, res, next) {
  if (!req.file) {
    return next();
  }

  const gcsname = Date.now() + req.file.originalname;
  const file = bucket.file(gcsname);

  const stream = file.createWriteStream({
    metadata: {
      contentType: req.file.mimetype
    },
    resumable: false
  });

  stream.on('error', (err) => {
    req.file.cloudStorageError = err;
    next(err);
  });

  stream.on('finish', () => {
    req.file.cloudStorageObject = gcsname;
    file.makePublic().then(() => {
      req.file.cloudStoragePublicUrl = getPublicUrl(gcsname);
      next();
    });
  });

  stream.end(req.file.buffer);
}
1
votes

@doug-stevenson thanks for pushing me in the right direction. I was able to get it to work with the following code:

const { Storage } = require('@google-cloud/storage');
const storage = new Storage();
const bucketName = 'test_bucket';
const blobName = 'test.csv';
const bucket = storage.bucket(bucketName);
const blob = bucket.file(blobName);
const request = require('request');


function pipeCSVToGCS(redirectUrl) {
      request.get(redirectUrl)
      .pipe(blob.createWriteStream({
          metadata: {
              contentType: 'text/csv'
          }
      }))
    .on("error", (err) => {
        console.error(`error occurred`);
    })
    .on('finish', () => {
        console.info(`success`);
    });
};