Azure Data Factory prohibit Pipeline Double Run

Question

I know it might be a bit a confusing title but couldn't get up to anythig better. The problem ... I have a ADF Pipeline with 3 Activities, first a Copy to a DB, then 2 times a Stored procedure. All are triggered by day and use a WindowEnd to read the right directory or pass a data to the SP.

There is no way I can get a import-date into the XML files that we are receiving.

So i'm trying to add it in the first SP. Problem is that once the first action from the pipeline is done 2 others are started.

The 2nd action in the same slice, being the SP that adds the dates, but in case history is loaded the same Pipeline starts again a copy for another slice.

So i'm getting mixed up data. As you can see in the 'Last Attempt Start'. Anybody has a idea on how to avoid this ?

ADF Monitoring

Answer 1

In case somebody hits a similar problem.. I've solved the problem by working with daily named tables. each slice puts its data into a staging table with a _YYYYMMDD after, can be set as"tableName": "$$Text.Format('[stg].[filesin_1_{0:yyyyMMdd}]', SliceEnd)". So now there is never a problem anymore of parallelism. The only disadvantage is that the SP's coming after this first have to work with Dynamic SQL as the table name where they are selecting from is variable. But that wasn't a big coding problem. Works like a charm !