Can't Download File From Cloud Storage to Cloud Function Temporary Storage to Read and Process CSV Import

Question

I am trying to use a firebase onCall funtion to download a CSV file from Cloud Storage, place it in cloud function temporary storage, then read it and upload it into a firestore collection called test. Unfortunately, I can't get the download to cloud function temporary storage to work.

I have tried to follow the google documentation at https://firebase.google.com/docs/storage/extend-with-functions and other examples with the "return bucket.file(filePath).download({destination: tempFilePath}), but without luck.

Here is my code

//THIS IS THE FUNCTION FOR LOADING AN PROCESSING A INCOMING CSV FILE
exports.uploadCSV = functions.https.onCall((data,context)=> {

    //get the file name from the boday
    const fileOne = data.id + '.csv';
        console.log('fileOne', fileOne);

    //bring together the director and file name
    const fileName = 'uploads/' + fileOne;
        console.log('fileName', fileName);

    //setup the temporary path
    const tmpFilePath = path.join(os.tmpdir(), fileName);
        console.log('tmpFilePath', tmpFilePath);

    //assign the Firbase project id
    const projectId = 'xxxxxxxxx';
        console.log('projectId')

    //assign storage to the project
    const storage = new Storage({ projectId: projectId });
        console.log('new Storage')

    //setup the bucket configuration
    const bucket = admin.storage().bucket();
        console.log('bucket', bucket)

    //Reference Storage Bucket in firestore
    const bucketName = 'xxxxxxxxx.appspot.com';
        console.log('bucketName');

    //sets myFile to value of file downloaded into cloud function temporary storage.
    return storage.bucket(bucketName).file(fileName).download({destination: tmpFilePath})
        .then(() => {
            console.log('image downloaded to:', tmpFilePath)
                    //read stream and process
            fs.createReadStream('tmpFilePath')
                .pipe(csv())
                .on('data', (row) => {
                    console.log(row);
                    if(row) {
                        firestore.collection('test').add({
                            name: row.Name
                        });
                    } else {console.log('no data')
                    }
                })
                .on('end', () => {
                    console.log('CSV file successfully processed.');
                });

        })
        .then(() => {
                //removes temporary directory
                return fs.unlink(tmpFilePath);
        })
        .catch((err)=>{ console.log(err); });

});

While the Network console returns a 200, this is the error I receive in my firebase-functions logs.

{ Error: ENOENT: no such file or directory, open '/tmp/uploads/somedata.csv' errno: -2, code: 'ENOENT', syscall: 'open', path: '/tmp/uploads/somedata.csv' }

Doug Stevenson Doug Stevenson · Accepted Answer · 2019-04-09T23:04:09

Your download path includes a folder called "uploads", but you never create that directory explicitly. Instead, try just downloading to os.tmpdir() without any intermediate directories to create:

// no "uploads" here
const fileName = fileOne;
    console.log('fileName', fileName);

const tmpFilePath = path.join(os.tmpdir(), fileName);
    console.log('tmpFilePath', tmpFilePath);

Can't Download File From Cloud Storage to Cloud Function Temporary Storage to Read and Process CSV Import

1 Answers