0
votes

This is a 2 part issue:

  1. How to move/delete a file after processing. Currently we can only copy, but the original file remains in the source. Ideally we don't want to add a separate process to get metadata and compare if processed or not. I have been following one blog that asks to use a WEB activity and the Delete Rest API. I have gotten to that point, but I am not able to understand/follow instructions on to obtain the ACCESS TOKEN. It shows Curl steps, if someone can help on this would be great.
  2. I have created a pipeline, that has 4 Main activities. As part of this pipeline I want to be able to send email notifications for Success and failure.

Each success of an activity moves to the next activity, but if any activity fails it will send email. I Want to be able to have one SUCCESS or FAILURE Notification and dynamically add content to email instead of sending email on previous activity. For Failure it seems like I have to create a separate web activity for each activity to align a Failure, which is not elegant.

Based on how the WEB activity is set to use the logical apps for email, we have to define the activity name that the email is associated with (at least based on my understanding). So i created a separate WEB activity for each failure email, not very elegant.

Does anyone have a better way to handle.

enter image description here

enter image description here

1
There is now a delete activity in ADF that works on blob, ADLS Gen1, and ADLS Gen2! docs.microsoft.com/en-us/azure/data-factory/delete-activity - mmarie

1 Answers

0
votes

The only alternative is using a logic app to delete blobs, you can trigger the logic app also via a Web activity. Can you share the blog URL for the delete via web?

As for the email on failure, the issue is that you cannot have 'or' dependencies in the pipeline. A workaround that I'm using is to introduce a variable, when an activity fails or when the pipeline is finished, it sets the variable. In parallel in the pipeline I have a while activity that checks this variable. When it gets a certain value an action will be triggered. Also not elegant, but you don't need a lot of Web activities this way.