Azure Data Factory - runs all the time?

0
votes

I've implemented a Azure DF Job which executes a SQL Stored Proc:

{
    "name": "spLoggingProc",
    "properties": {
        "activities": [
            {
                "type": "SqlServerStoredProcedure",
                "typeProperties": {
                    "storedProcedureName": "logging"
                },
                "outputs": [
                    {
                        "name": "spEmptyOutput15-4"
                    }
                ],
                "scheduler": {
                    "frequency": "Hour",
                    "interval": 1
                },
                "name": "spLogging"
            }
        ],
        "start": "2017-01-01T00:00:00Z",
        "end": "2099-01-01T00:10:00Z",
        "isPaused": false,
        "hubName": "dwh_hub",
        "pipelineMode": "Scheduled"
    }
}

The dataset:

{
    "name": "spEmptyOutput15-4",
    "properties": {
        "published": false,
        "type": "AzureSqlTable",
        "linkedServiceName": "DWH",
        "typeProperties": {
            "tableName": "spEmptyOutput15-4"
        },
        "availability": {
            "frequency": "Hour",
            "interval": 1
        }
    }
}

The problem is now, the Proc runs every 2-3 seconds. But frequency is set to every hour. My goal is, to run every hour and every day the proc.

Can anyone please help me?

Thanks a lot!

2
azureazure-data-factory

2 Answers

0
votes

Please change the start time to today's date and you will not see the issue. Because you have set the start time to start of year, it will run for each day and each hour so it keep on running for 24x166 times before coming to normal routine. Its still running on hourly basis but it has to complete the past runs for each hour, you will see that its running every few seconds. I am sure that your proc is just taking 1-2 seconds to complete.

0
votes

There is another way to run 10 slices (10 is Maximum value) parallelly to increase the rate. If you want the past data also. Then this will be helpful.

Change the Concurrency value 3 under Policy to run the slices in parallel. 

        "policy": {
        "concurrency": 3, 
        "executionPriorityOrder": "OldestFirst",
        "retry": 3,
        "timeout": "00:10:00" 
         }