I am following the tutorial Implementing Terraform on Microsoft Azure to the letter. So far so good and I have reached the module where terraform workspaces are used to create environments from the Azure DevOps CI/CD pipeline. The module name is "Using Azure DevOps"
The Azure DevOps project is public - https://dev.azure.com/MarkKharitonov0271/_git/Globomantics-testing
Running the workspaces release pipeline consistently fails for me:
2020-01-19T03:23:20.5420275Z Error: authorization.RoleDefinitionsClient#CreateOrUpdate: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client '0e648d2d-a49f-407e-99de-9d6343876a8c' with object id '0e648d2d-a49f-407e-99de-9d6343876a8c' does not have authorization to perform action 'Microsoft.Authorization/roleDefinitions/write' over scope '/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3' or the scope is invalid. If access was recently granted, please refresh your credentials."
2020-01-19T03:23:20.5420512Z
2020-01-19T03:23:20.5420989Z on vnet-peering.tf line 52, in resource "azurerm_role_definition" "vnet-peering":
2020-01-19T03:23:20.5421339Z 52: resource "azurerm_role_definition" "vnet-peering" {
(You can view the vnet-peering.tf in the project repo - https://dev.azure.com/MarkKharitonov0271/_git/Globomantics-testing?path=%2Fnetworking%2Fvnet-peering.tf)
Anyway, if I understand the error message correctly, it claims that the Service Principal 0e648d2d-a49f-407e-99de-9d6343876a8c does not have the rights to create a new role definition in the subscription 2b38509c-a310-4c8f-bd78-9e400cc874e3
So, I checked the Service Principal:
PS /home/mark> az ad sp show --id '0e648d2d-a49f-407e-99de-9d6343876a8c'
{
"accountEnabled": "True",
"addIns": [],
"alternativeNames": [],
"appDisplayName": "MarkKharitonov0271-Globomantics-testing-2b38509c-a310-4c8f-bd78-9e400cc874e3",
"appId": "0ae4ffc7-149d-45ac-ab15-c9f61e4591f8",
"appOwnerTenantId": "717e5a4d-529c-4ab2-a1c5-6a5f6345d8e4",
"appRoleAssignmentRequired": false,
"appRoles": [],
"applicationTemplateId": null,
"deletionTimestamp": null,
"displayName": "MarkKharitonov0271-Globomantics-testing-2b38509c-a310-4c8f-bd78-9e400cc874e3",
"errorUrl": null,
"homepage": "https://VisualStudio/SPN",
"informationalUrls": {
"marketing": null,
"privacy": null,
"support": null,
"termsOfService": null
},
"keyCredentials": [],
"logoutUrl": null,
"notificationEmailAddresses": [],
"oauth2Permissions": [
{
"adminConsentDescription": "Allow the application to access MarkKharitonov0271-Globomantics-testing-2b38509c-a310-4c8f-bd78-9e400cc874e3 on behalf of the signed-in user.",
"adminConsentDisplayName": "Access MarkKharitonov0271-Globomantics-testing-2b38509c-a310-4c8f-bd78-9e400cc874e3",
"id": "d0f141b9-fc6b-4f3c-9217-018d74712ee1",
"isEnabled": true,
"userConsentDescription": "Allow the application to access MarkKharitonov0271-Globomantics-testing-2b38509c-a310-4c8f-bd78-9e400cc874e3 on your behalf.",
"userConsentDisplayName": "Access MarkKharitonov0271-Globomantics-testing-2b38509c-a310-4c8f-bd78-9e400cc874e3",
}
],
"objectId": "0e648d2d-a49f-407e-99de-9d6343876a8c",
"objectType": "ServicePrincipal",
"odata.metadata": "https://graph.windows.net/717e5a4d-529c-4ab2-a1c5-6a5f6345d8e4/$metadata#directoryObjects/@Element",
"odata.type": "Microsoft.DirectoryServices.ServicePrincipal",
"passwordCredentials": [],
"preferredSingleSignOnMode": null,
"preferredTokenSigningKeyEndDateTime": null,
"preferredTokenSigningKeyThumbprint": null,
"publisherName": "Default Directory",
"replyUrls": [
"https://VisualStudio/SPN"
],
"samlMetadataUrl": null,
"samlSingleSignOnSettings": null,
"servicePrincipalNames": [
"https://VisualStudio/SPN136d4f76-7262-4ab0-8fbb-7be74dfc803b",
"0ae4ffc7-149d-45ac-ab15-c9f61e4591f8"
],
"servicePrincipalType": "Application",
"signInAudience": "AzureADMyOrg",
"tags": [],
"tokenEncryptionKeyId": null
}
Notice that the Service Principal has appId equal to 0ae4ffc7-149d-45ac-ab15-c9f61e4591f8.
And it seems to correspond to the one created by Azure DevOps when I added the Terraform tasks to the pipeline when it wanted to authorize access to the subscription. Indeed:
Now, the terraform apply step references the same service principal:
Finally, this Service Principal seems to have Contributor access to the subscription:
PS /home/mark> az role assignment list --assignee '0e648d2d-a49f-407e-99de-9d6343876a8c'
[
{
"canDelegate": null,
"id": "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/providers/Microsoft.Authorization/roleAssignments/346f1b92-0621-44c0-b88a-343c52637a0f",
"name": "346f1b92-0621-44c0-b88a-343c52637a0f",
"principalId": "0e648d2d-a49f-407e-99de-9d6343876a8c",
"principalName": "https://VisualStudio/SPN136d4f76-7262-4ab0-8fbb-7be74dfc803b",
"principalType": "ServicePrincipal",
"roleDefinitionId": "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c",
"roleDefinitionName": "Contributor",
"scope": "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3",
"type": "Microsoft.Authorization/roleAssignments"
}
]
PS /home/mark>
So, I do not understand what is going on.
EDIT 1
So, I was able to run exactly the same terraform configuration with exactly the same Service Principal from the Azure Cloud Shell.
First I reset the credentials, logged out and logged in back as that Service Principal:
PS /home/mark/terraform> az ad sp credential reset --name https://VisualStudio/SPN136d4f76-7262-4ab0-8fbb-7be74dfc803b
{
"appId": "0ae4ffc7-149d-45ac-ab15-c9f61e4591f8",
"name": "https://VisualStudio/SPN136d4f76-7262-4ab0-8fbb-7be74dfc803b",
"password": "e...3",
"tenant": "717e5a4d-529c-4ab2-a1c5-6a5f6345d8e4"
}
PS /home/mark/terraform> az account clear
Logout successful. Re-login to your initial Cloud Shell identity with 'az login --identity'. Login with a new identity with 'az login'.
PS /home/mark/terraform> az login --service-principal -u https://VisualStudio/SPN136d4f76-7262-4ab0-8fbb-7be74dfc803b -p e...3 --tenant 717e5a4d-529c-4ab2-a1c5-6a5f6345d8e4
Cloud Shell is automatically authenticated under the initial account signed-in with. Run 'az login' only if you need to use a different account
{
"cloudName": "AzureCloud",
"id": "2b38509c-a310-4c8f-bd78-9e400cc874e3",
"isDefault": true,
"name": "Visual Studio Enterprise",
"state": "Enabled",
"type": "servicePrincipal"
}
}
]
Then I cloned the Git repository to obtain the Terraform code and copied over the backend configuration saved from the previous modules:
PS /home/mark/terraform> git clone https://[email protected]/MarkKharitonov0271/Globomantics-testing/_git/Globomantics-testing
Cloning into 'Globomantics-testing'...
remote: Azure Repos
remote: We noticed you're using an older version of Git. For the best experience, upgrade to a newer version.
remote: Found 9 objects to send. (2 ms)
Unpacking objects: 100% (9/9), done.
Checking connectivity... done.
PS /home/mark/terraform> cd ./Globomantics-testing/networking/
PS /home/mark/terraform/Globomantics-testing/networking> dir
Directory: /home/mark/terraform/Globomantics-testing/networking
Mode LastWriteTime Length Name
---- ------------- ------ ----
------ 1/19/20 4:41 AM 40 backend.tf
------ 1/19/20 4:41 AM 2256 main.tf
------ 1/19/20 4:41 AM 436 terraform.tfvars
------ 1/19/20 4:41 AM 2423 vnet-peering.tf
------ 1/19/20 4:41 AM 279 workspacetest.sh
PS /home/mark/terraform/Globomantics-testing/networking> copy ../../1-main-vnet/backend-config.txt .
Now I am ready to initialize Terraform and set the development workspace:
PS /home/mark/terraform/Globomantics-testing/networking> terraform init -backend-config='backend-config.txt'
Initializing modules...
Downloading Azure/vnet/azurerm 1.2.0 for vnet-main...
- vnet-main in .terraform/modules/vnet-main
Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "azurerm" (hashicorp/azurerm) 1.41.0...
- Downloading plugin for provider "template" (hashicorp/template) 2.1.2...
...
PS /home/mark/terraform/Globomantics-testing/networking> terraform workspace select development
Switched to workspace "development".
Now terraform plan:
PS /home/mark/terraform/Globomantics-testing/networking> terraform plan -var sec_client_secret='F...$' -out main.tfplan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.template_file.subnet_prefixes[0]: Refreshing state...
data.template_file.subnet_prefixes[1]: Refreshing state...
data.template_file.subnet_prefixes[2]: Refreshing state...
azurerm_resource_group.main: Refreshing state... [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet]
data.azurerm_subscription.current: Refreshing state...
module.vnet-main.azurerm_resource_group.vnet: Refreshing state... [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet]
module.vnet-main.azurerm_virtual_network.vnet: Refreshing state... [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet]
module.vnet-main.azurerm_subnet.subnet[0]: Refreshing state... [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet/subnets/web]
module.vnet-main.azurerm_subnet.subnet[2]: Refreshing state... [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet/subnets/app]
module.vnet-main.azurerm_subnet.subnet[1]: Refreshing state... [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet/subnets/database]
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# azurerm_role_assignment.vnet will be created
+ resource "azurerm_role_assignment" "vnet" {
+ id = (known after apply)
+ name = (known after apply)
+ principal_id = "63309b8d-908a-4dcf-b95b-25eae33aaceb"
+ principal_type = (known after apply)
+ role_definition_id = (known after apply)
+ role_definition_name = (known after apply)
+ scope = "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet"
+ skip_service_principal_aad_check = (known after apply)
}
# azurerm_role_definition.vnet-peering will be created
+ resource "azurerm_role_definition" "vnet-peering" {
+ assignable_scopes = [
+ "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3",
]
+ id = (known after apply)
+ name = "allow-vnet-peer-action-development"
+ role_definition_id = (known after apply)
+ scope = "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3"
+ permissions {
+ actions = [
+ "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write",
+ "Microsoft.Network/virtualNetworks/peer/action",
+ "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read",
+ "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete",
]
+ not_actions = []
}
}
# azurerm_virtual_network_peering.main will be created
+ resource "azurerm_virtual_network_peering" "main" {
+ allow_forwarded_traffic = (known after apply)
+ allow_gateway_transit = (known after apply)
+ allow_virtual_network_access = (known after apply)
+ id = (known after apply)
+ name = "development_2_sec"
+ remote_virtual_network_id = "/subscriptions/2b1285d1-f4a7-4cc3-a7b1-0d0fc31d6192/resourceGroups/security/providers/Microsoft.Network/virtualNetworks/security"
+ resource_group_name = "development-vnet"
+ use_remote_gateways = (known after apply)
+ virtual_network_name = "development-vnet"
}
# azurerm_virtual_network_peering.sec will be created
+ resource "azurerm_virtual_network_peering" "sec" {
+ allow_forwarded_traffic = (known after apply)
+ allow_gateway_transit = (known after apply)
+ allow_virtual_network_access = (known after apply)
+ id = (known after apply)
+ name = "sec_2_development"
+ remote_virtual_network_id = "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet"
+ resource_group_name = "security"
+ use_remote_gateways = (known after apply)
+ virtual_network_name = "security"
}
Plan: 4 to add, 0 to change, 0 to destroy.
...
------------------------------------------------------------------------
This plan was saved to: main.tfplan
To perform exactly these actions, run the following command to apply:
terraform apply "main.tfplan"
And finally terraform apply:
PS /home/mark/terraform/Globomantics-testing/networking> terraform apply main.tfplan
azurerm_role_definition.vnet-peering: Creating...
azurerm_role_definition.vnet-peering: Creation complete after 1s [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/providers/Microsoft.Authorization/roleDefinitions/b9614597-de13-c2c6-a275-9186847642ed]
azurerm_role_assignment.vnet: Creating...
azurerm_role_assignment.vnet: Creation complete after 1s [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet/providers/Microsoft.Authorization/roleAssignments/7f6a642d-d43e-05b5-cf34-3d20d02cd255]
azurerm_virtual_network_peering.main: Creating...
azurerm_virtual_network_peering.sec: Creating...
azurerm_virtual_network_peering.main: Still creating... [10s elapsed]
azurerm_virtual_network_peering.sec: Still creating... [10s elapsed]
azurerm_virtual_network_peering.main: Creation complete after 11s [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet/virtualNetworkPeerings/development_2_sec]
azurerm_virtual_network_peering.sec: Creation complete after 11s [id=/subscriptions/2b1285d1-f4a7-4cc3-a7b1-0d0fc31d6192/resourceGroups/security/providers/Microsoft.Network/virtualNetworks/security/virtualNetworkPeerings/sec_2_development]
...
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
Outputs:
resource_group_name = development-vnet
vnet_id = /subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet
vnet_name = development-vnet
And it all works fine using exactly the same Service Principal that was created by Azure DevOps.
I will try next to delete the Service Connection on the Azure DevOps project, create a new one manually and retry the release.
EDIT 2
Same error with the new Service Connection. I do not understand it. What I find "curious" is that all the resources in main.tf were provisioned just fine, but none in vnet-peering.tf.
What is going on here?
EDIT 3
Replaced the Terraform tasks by Microsoft DevLabs with those by Charles Zipp, because the Pluralsight course author used them. Same result.
