How to mention .csv and .json file format in config.json file in cloud functions to trigger DAG

Question

I am using Cloud functions to trigger Airflow DAG when ever the file is placed in the cloud storage.

It was working for .csv file but my requirement is cloud function should trigger DAG when any type of file (say .json file) is placed in the cloud storage.

I have given configuration like,

index.js:

  'use strict';

   const fetch = require('node-fetch');
    const FormData = require('form-data');

    var config = require('./config.json');

   exports.triggerGCSDag = function triggerGCSDag(data,context) {

   const PROJECT_ID = config.PROJECT_ID;
   const CLIENT_ID = config.CLIENT_ID;
   const WEBSERVER_ID = config.WEBSERVER_ID;

   const USER_AGENT =config.USER_AGENT;
   const BODY = {'conf': JSON.stringify(data)};
   const file = data;

   const file_format=config.file_format;  
   const folder_array=config.folder_name;
   const DAG_ARRAY=config.DAG_NAME; 
   .
   .
   .

But DAG is not triggered whenever .json or csv file placed in the cloud storage.

Please help me in out to set config.json for file format csv and json so that cf will trigger Airflow DAG

You have not shown any code that does anything. Edit your question and include the code that you have written. This link will help you create your question: stackoverflow.com/help/minimal-reproducible-example — John Hanley

Chris32 Chris32 · Accepted Answer · 2019-10-31T19:52:51

You can use the event finalize as a trigger for your function, then you can use the trigger data to determine if the uploaded file is a .csv or .json or any other type of file. You can see how to use the trigger data in the Storage Triggers documentation.

You can use this function as an example.

This function responds with "goog" if the file has the desired extension, and with "not goog" if have any other extension. You can find the response in the logs of your function.

Now you just need to add your logic to trigger Airflow inside of the conditional.

  exports.helloGCS = (data, context) => {
  const file = data;
  console.log(`File: ${file.name}`);
  if(file.name.includes('.csv') || file.name.includes('.json')){
  console.log('goog');
  } else {
   console.log('not goog');
  }
  };

How to mention .csv and .json file format in config.json file in cloud functions to trigger DAG

index.js:

1 Answers