Azure Data Factory Release Pipeline - Resource Not Found Error

1
votes

I am testing a build/release of a very simple ADF (just one activity), the build, repository, arm template export all seem ok until when I run the release task, the error shows up in the final step, that says:

 "error": {
    "code": "ResourceNotFound",
    "message": "The Resource 'Microsoft.DataFactory/factories/htTestDF' under resource group 'xxx-rg' was not found."
  }

I watched several tutorials and microsoft web sites, and tried exporting the ARM template several times, the same error occurs. Any ideas will be greatly appreciated.

2
azureazure-pipelines-release-pipeline
share the template, please – 4c74356b41
This means the factory you are using does not exist under resource group 'xxx-rg' now. Have you check it before you release this template?You'd better share your template and the screenshots about the ADF. – Merlin Liang - MSFT
Hi I will upload the template shortly. Perhaps I am not understanding the concept. I am getting the same error consistently either doing export/import Template for Data Factory or testing the Azure DevOps Release Pipeline. I have a data factory in a source environment, it is with a different resource group than the tartget env. When I import into the target env, I substitute the resource group parm to the target resource group. It fails. Am I simulating it incorrectly? – HT1
Here is my arm_template.json: drive.google.com/drive/folders/… – HT1
When importing the template on the target environment side, I used "Deploy Custom Template", and loaded this template file. The wizard lets me specify the target environment resource group. After that, when I accept the wizard, the processing will error out with: { "error": { "code": "ResourceNotFound", "message": "The Resource 'Microsoft.DataFactory/factories/HTDF3' under resource group 'slalom_rg' was not found." } – HT1

2 Answers

2
votes

Thanks for your details clarifies in comments. Now, the error message you met caused by using a different target resource group as this ARM template deploy to.

To make this more clear, I reproduce the issue based on the details you provided. Fortunately, got the same error with you. Now, let's focus on its log, then get why it cause the Not Found error.(Please set debug=true)

enter image description here

As I mentioned in the above pic, it is the api that this task used at first step while the template begin to apply into the corresponding resource group and deployment. For more cleared, please refer to this REST API doc firstly: Deployments - Create Or Update.

The logic of this task is compile parameters from ARM template file, pack them and use it as request body for this PUT api call. See its api doc, you can get that for this API call, its resourceGroupName and deploymentName need to be specified firstly. In another words, if you specified another target resource group, it would not find the correct target place that can apply this template definition. Because, you can see that this ARM template is preparing to applying activities SetVariable into your Data Factory HTDF3 and the defined pipelines name is HTPipe1. But these should all not exists in your target Resource Group. Thus it caused the error like this:

"error": {
    "code": "ResourceNotFound",
    "message": "The Resource 'Microsoft.DataFactory/factories/Merlin-1003' under resource group 'Merlin-ARM-deploy' was not found."
  }

In my sample, Merlin-ARM-deploy is my target resource group.

If you want to deploy this into your target resource group, you need to create one data factory manually, or use another ARM Template to create a new one same factory in the target resource group. If you choose the previous method, just then modify the template.json file, to let its parameters correspond to the actual target resource group. But if use the second method, do not operate anything. Just apply them with task.

0
votes

The ARM template generated by ADF(publish) cannot be deployed directly to a new RG.

Solution

  1. Create RG(optional, assuming it is IAC(infra as code))

  2. Run a powershell script task in pipeline to create an empty ADF(do not use empty ARM instead). Since it is not ARM you would need to put an optional condition to check if it already exists)

    Set-AzDataFactoryV2 -ResourceGroupName "RG" -Name "ADF" -Location "North Europe"

  3. Now we can execute the ARM template from publish folder (the one you had given)

Errors in ADF publish system.

  1. The ARM template need to be generated in such a way that it is idempotent(should also create afresh if not present). But it is not at the moment. It expects an ADF to be present already(strangely).
  2. When an empty ARM template created in another RG is used to create a fresh empty ADF in this RG(newly created), it fails. Well, it creates empty ADF but we cannot put adf_publish(default publish folder for ADF) on top of it because we get 'resource not found error'.
  3. But when we manually create an ADF and run the adf_publish template then it works! But ofcourse, this is not what we want.
  4. Why does manual & powershell work(empty + publish) but not ARM template? It could be that the ARM Template has wrong location/region mentioned in it but that was not the case.(really puzzling to me)