Terraform - Rescource group already exists even though part of backend with terraform apply

Question

I've extended my terraform definitions which I run via an Azure pipeline. The terraform init task has a backend configuration on the pipeline which defines the resource group medquality-aks but it seems there is an issue. I am not even sure though since there are 3 errors at once.

Earlier I thought there was a permission issue so I added the create permission to the Microsoft Graph API for terraform.

The resource group does not exist before running the pipeline though.

2020-12-30T20:30:05.2720640Z [0m[1mrandom_id.log_analytics_workspace_name_suffix: Creating...[0m[0m
2020-12-30T20:30:05.2726774Z [0m[1mrandom_string.sp-aks-password: Creating...[0m[0m
2020-12-30T20:30:05.2763813Z [0m[1mrandom_string.sp-aks-secret: Creating...[0m[0m
2020-12-30T20:30:05.2769915Z [0m[1mrandom_string.sp-aks-password: Creation complete after 0s [id=i<50H)EbQENB::#nf>Z4Af@k][0m[0m
2020-12-30T20:30:05.2786026Z [0m[1mrandom_id.log_analytics_workspace_name_suffix: Creation complete after 0s [id=qByINuQDrlg][0m[0m
2020-12-30T20:30:05.2806341Z [0m[1mrandom_string.sp-aks-secret: Creation complete after 0s [id=kLLI(=aZ4K[+%kN[%xay-E9h][0m[0m
2020-12-30T20:30:05.5417261Z [0m[1mazuread_group.aks_administrators: Creating...[0m[0m
2020-12-30T20:30:05.5463857Z [0m[1mazuread_application.sp-aks: Creating...[0m[0m
2020-12-30T20:30:09.4043339Z [0m[1mazurerm_resource_group.mq-aks: Creating...[0m[0m
2020-12-30T20:30:09.4519178Z [31m
2020-12-30T20:30:09.4521407Z [1m[31mError: [0m[0m[1mgraphrbac.ApplicationsClient#Create: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Unknown" Message="Unknown service error" Details=[{"odata.error":{"code":"Authorization_RequestDenied","date":"2020-12-30T20:30:05","message":{"lang":"en","value":"Insufficient privileges to complete the operation."},"requestId":"27125e18-2100-43b3-970b-9be30bed427e"}}][0m
2020-12-30T20:30:09.4522796Z 
2020-12-30T20:30:09.4523723Z [0m  on aks-ad-sp.tf line 11, in resource "azuread_application" "sp-aks":
2020-12-30T20:30:09.4524357Z   11: resource "azuread_application" "sp-aks" [4m{[0m
2020-12-30T20:30:09.4524725Z [0m
2020-12-30T20:30:09.4524971Z [0m[0m
2020-12-30T20:30:09.4525228Z [31m
2020-12-30T20:30:09.4526780Z [1m[31mError: [0m[0m[1mcreating Group ("aks-administrators"): graphrbac.GroupsClient#Create: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Unknown" Message="Unknown service error" Details=[{"odata.error":{"code":"Authorization_RequestDenied","date":"2020-12-30T20:30:05","message":{"lang":"en","value":"Insufficient privileges to complete the operation."},"requestId":"30882842-8926-4eee-8d45-847759849087"}}][0m
2020-12-30T20:30:09.4528021Z 
2020-12-30T20:30:09.4528543Z [0m  on aks-administrators-group.tf line 1, in resource "azuread_group" "aks_administrators":
2020-12-30T20:30:09.4529197Z    1: resource "azuread_group" "aks_administrators" [4m{[0m
2020-12-30T20:30:09.4529531Z [0m
2020-12-30T20:30:09.4529922Z [0m[0m
2020-12-30T20:30:09.4530174Z [31m
2020-12-30T20:30:09.4531284Z [1m[31mError: [0m[0m[1mA resource with the ID "/subscriptions/ae250472-5313-4abf-a081-3f746e68c88f/resourceGroups/medquality-aks" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_resource_group" for more information.[0m
2020-12-30T20:30:09.4532029Z 
2020-12-30T20:30:09.4532489Z [0m  on aks-rg.tf line 1, in resource "azurerm_resource_group" "mq-aks":
2020-12-30T20:30:09.4532999Z    1: resource "azurerm_resource_group" "mq-aks" [4m{[0m
2020-12-30T20:30:09.4533300Z [0m
2020-12-30T20:30:09.4533543Z [0m[0m
2020-12-30T20:30:09.4545723Z 
2020-12-30T20:30:09.4679091Z ##[error]Terraform command 'apply' failed with exit code '1'.:  graphrbac.ApplicationsClient#Create: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Unknown" Message="Unknown service error" Details=[{"odata.error":{"code":"Authorization_RequestDenied","date":"2020-12-30T20:30:05","message":{"lang":"en","value":"Insufficient privileges to complete the operation."},"requestId":"27125e18-2100-43b3-970b-9be30bed427e"}}] |  creating Group ("aks-administrators"): graphrbac.GroupsClient#Create: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Unknown" Message="Unknown service error" Details=[{"odata.error":{"code":"Authorization_RequestDenied","date":"2020-12-30T20:30:05","message":{"lang":"en","value":"Insufficient privileges to complete the operation."},"requestId":"30882842-8926-4eee-8d45-847759849087"}}] |  A resource with the ID "/subscriptions/ae250472-5313-4abf-a081-3f746e68c88f/resourceGroups/medquality-aks" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_resource_group" for more information.
2020-12-30T20:30:09.9560347Z ##[section]Finishing: terraform apply

The permissions I've currently set for the API:

Update:

After dealing with some issues on how to create a service principal I know have an issue on the role assignment provisioning, the previous issues seem fine now, is that a matter of policy roles? I didn't see anything in the hashicorp documentation though:

2021-01-05T02:32:19.0757008Z [0m[1mazurerm_kubernetes_cluster_node_pool.user: Creation complete after 4m10s [id=/subscriptions/ae250472-5313-4abf-a081-3f746e68c88f/resourcegroups/medquality-aks/providers/Microsoft.ContainerService/managedClusters/medquality-aks/agentPools/user][0m[0m
2021-01-05T02:32:19.0846822Z [31m
2021-01-05T02:32:19.0853022Z [1m[31mError: [0m[0m[1mauthorization.RoleAssignmentsClient#Create: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client '0595a82d-0ef5-4dce-a526-a348ad51ce6d' with object id '0595a82d-0ef5-4dce-a526-a348ad51ce6d' does not have authorization to perform action 'Microsoft.Authorization/roleAssignments/write' over scope '/subscriptions/ae250472-5313-4abf-a081-3f746e68c88f/resourceGroups/medquality-aks/providers/Microsoft.Network/virtualNetworks/medquality-aks-network/providers/Microsoft.Authorization/roleAssignments/c81c31c4-fa80-d98b-887f-b1d44852e7ce' or the scope is invalid. If access was recently granted, please refresh your credentials."[0m
2021-01-05T02:32:19.0855355Z 
2021-01-05T02:32:19.0856617Z [0m  on aks-rbac.tf line 1, in resource "azurerm_role_assignment" "sp-aks-network":
2021-01-05T02:32:19.0858454Z    1: resource "azurerm_role_assignment" "sp-aks-network" [4m{[0m
2021-01-05T02:32:19.0858855Z [0m
2021-01-05T02:32:19.0859142Z [0m[0m
2021-01-05T02:32:19.1005049Z 
2021-01-05T02:32:19.1063551Z ##[error]Terraform command 'apply' failed with exit code '1'.:  authorization.RoleAssignmentsClient#Create: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client '0595a82d-0ef5-4dce-a526-a348ad51ce6d' with object id '0595a82d-0ef5-4dce-a526-a348ad51ce6d' does not have authorization to perform action 'Microsoft.Authorization/roleAssignments/write' over scope '/subscriptions/ae250472-5313-4abf-a081-3f746e68c88f/resourceGroups/medquality-aks/providers/Microsoft.Network/virtualNetworks/medquality-aks-network/providers/Microsoft.Authorization/roleAssignments/c81c31c4-fa80-d98b-887f-b1d44852e7ce' or the scope is invalid. If access was recently granted, please refresh your credentials."
2021-01-05T02:32:19.4811396Z ##[section]Finishing: terraform apply

Answer 1

##[error]Terraform command 'apply' failed with exit code '1'.: graphrbac.ApplicationsClient#Create: Failure responding to request: StatusCode=403

According to the error information it indicated that you have no permission to do that.

You could refer to this doc about terraform permission:

If you're authenticating using a Service Principal then it must have permissions to both Read and write all (or owned by) applications and Sign in and read user profile within the Windows Azure Active Directory API.

You need to grant the Azure Active Directory Graph premission instead of Microsoft Graph permission.

You could navigate to Azure Active Directory -> App registrations -> Select the APP(used to create Azure Devops Service connection) -> API permissions -> Azure Active Directory Graph.

Delegated permissions -> Sign in and read user profile(User.Read)

Application permissions -> Read and write all (or owned by) applications (Application.ReadWrite.All) and Read and write directory data (Directory.ReadWrite.All)

In addition, you can also grant this App Contributor role in the subscription.

Here is the doc about the detailed steps to grant Contributor role.