Interact with Terraform resources behind firewall on Azure

Question

This challenge is regarding Azure and Azure DevOps but I would imagine this happening on similar platforms (AWS, GCP, Github, Gitlab, etc.)

I am currently using Azure DevOps Pipelines but I am facing a problem with interacting with resources behind firewalls (either IP restricted or virtual network restricted). As Azure Pipeline spins up a new VM it requires me to whitelist that given public IP for that newly spun up machine each time I do a run. It is very janky to accommodate for this whitelisting as I am creating Azure Pipelines as submodules for reproducibility purpose extending templates from one project and using it in multiple. Terraform state needs to access configurations on restricted resources, hence throwing access denied messages.

I have looked into the following to solve the challenge and my thoughts about them:

Implementing Azure Virtual Machine scale-set agents instead of Microsoft Hosted agent (https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops).
- This would require a long discussion with our security team as they are not fan of using virtual machines in the cloud.
Deploy Atlantis on AKS with a static public IP from the Load Balancer and whitelist this one IP (https://docs.microsoft.com/en-us/azure/aks/static-ip).
- This would require some initial setup from my side as I could not find that much documentation on deploying this using Azure (AKS) and Azure DevOps. (found this article: https://engineering.plex.com/posts/deploying-infrastructure-azure/ and this video using Github https://www.youtube.com/watch?v=33j49_n8Zvc&ab_channel=JJAsghar. Terraform Module for deploying Atlantis on AWS: https://github.com/terraform-aws-modules/terraform-aws-atlantis)
Use Terraform Enterprise for infrastructure deployment and whitelist Terraform Enterprise IP range (https://www.terraform.io/docs/cloud/api/ip-ranges.html).
- This would require introducing a new platform for provisioning infrastructure.
Make a huge whitelist stage that whitelists each resource from my Terraform Module Registry depending on the module used in the pipeline (very cumbersome to maintain).
- This would require too much maintenance and does not seem like the best solution.

What are your thoughts on solving this challenge?

If you are using hosted agent created in inside the network (or paired with) of the VM there should be no need for public ip? — magnarwium
I am using Microsoft Hosted Agents on Azure DevOps. This is considered a machine outside Azure just like any other machine (virtual or not). I need to allow for this machine to communicate to the restricted resources in the best possible manner. — Sebastian Balle
Maybe you could specify exactly which resources it is you need access to as "regular" terraform would mainly perform provisioning against the Azure Control plane and not necessarilly execute configuration on the resources themselves. There might be workarounds or other technologies that could help. — pijemcolu
It might be a container within a storage account which is restricted. — Sebastian Balle

Levi Lu-MSFT Levi Lu-MSFT · Accepted Answer · 2020-12-29T03:14:44

You can use scripts to get the ip of the cloud agents. And dynamically whitelist the ip address for your azure storage account using Azure PowerShel or Azure Cli. See below example:

1, Add Azure Powershell task before Terraform task in your azure devops pipeline to get the agent's ip address and add whitelist for azure storage account.

- task: AzurePowerShell@5
  displayName: 'Azure PowerShell script: InlineScript copy'
  inputs:
    azureSubscription: 'Microsoft-Azure'
    ScriptType: InlineScript
    Inline: |
     $ip = Invoke-RestMethod http://ipinfo.io/json | Select -exp ip #get agent ip
     #add ip to whitelist
     Add-AzStorageAccountNetworkRule -ResourceGroupName "myresourcegroup" -AccountName "mystorageaccount" -IPAddressOrRange $ip

    azurePowerShellVersion: LatestVersion

2, Add another azure powershell task at the end of your pipeline to remove the whitelist.

- task: AzurePowerShell@5
  displayName: 'Azure PowerShell script: InlineScript copy'
  inputs:
    azureSubscription: 'Microsoft-Azure'
    ScriptType: InlineScript
    Inline: |
     $ip = Invoke-RestMethod http://ipinfo.io/json | Select -exp ip
     
     Remove-AzStorageAccountNetworkRule -ResourceGroupName "myresourcegroup" -AccountName "mystorageaccount" -IPAddressOrRange $ip

    azurePowerShellVersion: LatestVersion

Check document here for more information.

The IP ranges for cloud agents changes weekly. You can also check the weekly file and update the whitelist ip address manually. Check here for more information.

Interact with Terraform resources behind firewall on Azure

1 Answers