Dataflow from Azure to Google Cloud Platform using NiFi

0
votes

My goal is to move json/xml files from Azure to Google Cloud Platform(GCP) using NiFi. After my all R&D I figured out few processors that could be helpful. Lists are below:

  1. GetFile
  2. PutFile
  3. putGCSObject- To put data in GCP
  4. Found this link as an alternative to get files from Azure since there is no in-built processor available

The above link was complicated. So according to my objective am I going right ?? Do I need any extra processors ?? and also any important configuration I need to make while defining this flow??

Please help me out since I am a newbie and have just started with NiFi

1
azurehadoopgoogle-cloud-platformbigdataapache-nifi

1 Answers

2
votes

If I get your question right, your objective is to move files that are in Azure File Storage to Google Cloud Storage. Correct?

If that's the case, then you are almost in the correct path. As you have mentioned, Apache NiFi doesn't have processors for Azure Files right now so follow the Microsoft guide which you have quoted here and mount Azure Files to your Linux machine and create the flow like this: ListFile -> FetchFile -> PutGCSObject

GetFile or any Get* processors were of legacy nature and the new List/Fetch pattern is quite good as it helps you with state tracking, etc.,

BTW, you don't need PutFile since the objective of that processor is to write file(s) into local filesystem but we are here dealing with GCS as our sink.