ADF ARM Template doesn't appear to include the factory itself

Question

I was in the process of configuring DevOps to deploy my Dev ADF to the UAT ADF instance.

I had come across the standard issue of the deploy not deleting out-dated pipelines, and attempted to use "Complete" deployment mode to resolve that.

Whereupon DevOps entirely deleted the UAT ADF instance!

Looking further at the docs, it appears that this is the expected behaviour if the factories are not in the ARM Templates. And looking at my ARM Template (generated entirely by ADF, and with [AFAIK] entirely standard settings), it confirms that the factory itself is NOT amongst the documented resources to be created.

This seems ... odd.

Am I missing something? How do I get the factory to be included in the ARM Template?

Or alternatively, how can I use the "Complete" deployment mode without it deleting the target ADF instance?

Note that the reason I don't want to use the "define a separate script to solve this" approach, is that it seems excessively complex when that the "Complete" mode sounds like it should do exactly what I want :) (If it weren't for this one oddity about deleting the factory)

DreadedFrost DreadedFrost · Accepted Answer · 2020-11-13T20:36:17

You are correct. I've run into this issue before. To circumnavigate it this I recommend creating a core ARM template that would contain the Data Factory and any necessary linked services solely used by Data Factory. This will ensure the "infrastructure/connections" are deployed when creating a new instance.

If you are following Azure Data Factory CI/CD this would be an additional Azure Resource Group Deployment task before the Pipelines are deployed and reference the ARM template which should be in a separate repository.

Here's a template for Data Factory w/ Log Analytics to get you started. I included Log Analytics as most people don't realize about Log retention until they need it. Plus it's a best practices. Just update the system name as this will create a naming standard of adf-systemName-environment-regionAbrviation. The region abbreviation is dynamic based upon the object and will look up agianst the resource group.

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "environment": {
            "type": "string",
            "metadata": "Name of the environment being deployed to"
        },
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]",
      "metadata": {
        "description": "Location for all resources."
      }
    }
    },
    "variables": {
        "systemName": "DataFactoryBaseName",
    "regionReference": {
      "centralus": "cus",
      "eastus": "eus",
      "westus": "wus"
    },
    "dataFactoryName": "[toLower(concat('adf-', variables('systemName'),'-', parameters('environment'),'-',variables('regionDeployment')))]",
    "logAnalyticsName": "[toLower(concat('law-', variables('systemName'),'-', parameters('environment'),'-',variables('regionDeployment')))]",
    "regionDeployment": "[toLower(variables('regionReference')[parameters('location')])]"
    },
    "resources": [
        {
  "name": "[variables('dataFactoryName')]",
  "type": "Microsoft.DataFactory/factories",
  "apiVersion": "2018-06-01",
  "location": "[parameters('location')]",
  "tags": {
        "displayName": "Data Factory",
        "ProjectName": "[variables('systemName')]",
        "Environment":"[parameters('environment')]"
  },
  "identity": {
    "type": "SystemAssigned"
  }
},
    {
      "type": "Microsoft.OperationalInsights/workspaces",
      "name": "[variables('logAnalyticsName')]",
      "tags": {
        "displayName": "Log Analytics",
        "ProjectName": "[variables('systemName')]",
        "Environment":"[parameters('environment')]"
      },
      "apiVersion": "2020-03-01-preview",
      "location": "[parameters('location')]"
    },
        {
      "type": "microsoft.datafactory/factories/providers/diagnosticsettings",
      "name": "[concat(variables('dataFactoryName'),'/Microsoft.Insights/diagnostics')]",
      "location": "[parameters('location')]",
      "apiVersion": "2017-05-01-preview",
      "dependsOn": [
        "[resourceID('Microsoft.OperationalInsights/workspaces',variables('logAnalyticsName'))]",
        "[resourceID('Microsoft.DataFactory/factories',variables('dataFactoryName'))]"
      ],
      "properties": {
        "name": "diagnostics",
        "workspaceId": "[resourceID('Microsoft.OperationalInsights/workspaces',variables('logAnalyticsName'))]",
        "logAnalyticsDestinationType": "Dedicated",
        "logs": [
          {
            "category": "PipelineRuns",
            "enabled": true,
            "retentionPolicy": {
              "enabled": false,
              "days": 0
            }
          },
          {
            "category": "TriggerRuns",
            "enabled": true,
            "retentionPolicy": {
              "enabled": false,
              "days": 0
            }
          },
          {
            "category": "ActivityRuns",
            "enabled": true,
            "retentionPolicy": {
              "enabled": false,
              "days": 0
            }
          }
        ],
        "metrics": [
          {
            "category": "AllMetrics",
            "timeGrain": "PT1M",
            "enabled": true,
            "retentionPolicy": {
              "enabled": false,
              "days": 0
            }
          }
        ]
      }
   }
    ]
}

ADF ARM Template doesn't appear to include the factory itself

1 Answers