Authenticating with Azure Repos git module sources in an Azure Pipelines build

2
votes

I'm currently creating a pipeline for Azure DevOps to validate and apply a Terraform configuration to different subscription.

My terraform configuration uses modules, those are "hosted" in other repositories in the same Azure DevOps Project as the terraform configuration.

Sadly, when I try to perform terraform init to fetch those modules, the pipeline task "hang" there waiting for credentials input.

As recommanded in the Pipeline Documentation on Running Git Commands in a script I tried to add a checkout step with the persistCredentials:true attribute.

From what I can see in the log of the task (see bellow), the credentials information are added specifically to the current repo and are not usable for other repos.

The command performed when adding persistCredentials:true

2018-10-22T14:06:54.4347764Z ##[command]git config http.https://[email protected]/my-org/my-project/_git/my-repo.extraheader "AUTHORIZATION: bearer ***"

The output of terraform init task

2018-10-22T14:09:24.1711473Z terraform init -input=false
2018-10-22T14:09:24.2761016Z Initializing modules...
2018-10-22T14:09:24.2783199Z - module.my-module
2018-10-22T14:09:24.2786455Z   Getting source "git::https://[email protected]/my-org/my-project/_git/my-module-repo?ref=1.0.2"

How can I setup the git credentials to work for other repositories ?

4
azure-devopsterraformazure-pipelinesazure-repos

4 Answers

2
votes

I had the same issue, what I ended up doing is tokenizing SYSTEM_ACCESSTOKEN in terraform configuration. I used Tokenzization task in Azure DevOps where __ prefix and suffix is used to identify and replace tokens with actual variables (it is customizable but I find double underscores best for not interfering with any code that I have)

- task: qetza.replacetokens.replacetokens-task.replacetokens@3
    displayName: 'Replace tokens'
    inputs:
      targetFiles: |
       **/*.tfvars
       **/*.tf
      tokenPrefix: '__'
      tokenSuffix: '__'

Something like find $(Build.SourcesDirectory)/ -type f -name 'main.tf' -exec sed -i 's~__SYSTEM_ACCESSTOKEN__~$(System.AccessToken)~g' {} \; would also work if you do not have ability to install custom extensions to your DevOps organization.

My terraform main.tf looks like this:

module "app" {
  source = "git::https://token:[email protected]/actualOrgName/actualProjectName/_git/TerraformModules//azure/app-service?ref=__app-service-module-ver__"
  ....
}

It's not beautiful but it gets the job done. Module source (at the time of writing) does not support variable input from terraform. So what we can do is to use Terrafile it's an open source project helping with keeping up with the modules and different versions of the same module you might use by keeping a simple YAML file next to your code. It seems that it's no longer being actively maintained, however it just works: https://github.com/coretech/terrafile my example of Terrafile:

app:
    source:  "https://token:[email protected]/actualOrgName/actualProjectName/_git/TerraformModules"
    version: "feature/handle-twitter"
app-stable:
    source:  "https://token:[email protected]/actualOrgName/actualProjectName/_git/TerraformModules"
    version: "1.0.5"

Terrafile by default download your modules to ./vendor directory so you can point your module source to something like:

module "app" {
  source = "./vendor/modules/app-stable/azure/app_service"
  ....
}

Now you just have to figure out how to execute terrafile command in the directory where Terrafile is present. My azure.pipelines.yml example:

- script: curl -L https://github.com/coretech/terrafile/releases/download/v0.6/terrafile_0.6_Linux_x86_64.tar.gz | tar xz -C $(Agent.ToolsDirectory)
  displayName: Install Terrafile

- script: |
    cd $(Build.Repository.LocalPath)
    $(Agent.ToolsDirectory)/terrafile
  displayName: Download required modules
1
votes

I did this

_ado_token.ps1

# used in Azure DevOps to allow terrform to auth with Azure DevOps GIT repos
$tfmodules = Get-ChildItem $PSScriptRoot -Recurse -Filter "*.tf"
foreach ($tfmodule in $tfmodules) {
    $content = [System.IO.File]::ReadAllText($tfmodule.FullName).Replace("git::https://myorg@","git::https://" + $env:SYSTEM_ACCESSTOKEN +"@")
    [System.IO.File]::WriteAllText($tfmodule.FullName, $content)
}

azure-pipelines.yml

- task: PowerShell@2
  env: 
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)
  inputs:
    filePath: '_ado_token.ps1'
    pwsh: true
  displayName: '_ado_token.ps1'
1
votes

You have essentially two ways of doing this.

Pre-requisite

Make sure that you read and, depending on your needs, that you apply the Enable scripts to run Git commands section from the "Run Git commands in a script" doc.

Solution #1: dynamically insert the System.AccessToken (or a PAT, but I would not recommend it) at pipeline runtime

You could to this either by:

  • inserting a replacement token such as __SYSTEM_ACCESSTOKEN__ in your code (as Nilsas suggests) and use some token replacement code or the qetza.replacetokens.replacetokens-task.replacetokens task to insert the value. The disadvantage of this solution is that you would also have to replace the token when you run you terraform locally.
  • using some code to replace all git::https://dev.azure.com text with git::https://[email protected].

I used the second approach by using the following bash task script (it searches terragrunt files but you can adapt to terraform files without much change):

- bash: |
    find $(Build.SourcesDirectory)/ -type f -name 'terragrunt.hcl' -exec sed -i 's~git::https://dev.azure.com~git::https://$(System.AccessToken)@dev.azure.com~g' {} \;

Abu Belai offers a PowerShell script to do something similar.

This type of solution does not however work if modules in your terraform modules git repo call themselves modules in another git repo, which was our case.

Solution #2: adding globally the access token in the extraheader of the url of your terraform modules git repos

This way, all the modules' repos, called directly by your code or called indirectly by the called modules' code, will be able to use your access token. I did so by adding the following step before your terraform/terragrunt calls:

- bash: |
    git config --global http.https://dev.azure.com/<your-org>/<your-first-repo-project>/_git/<your-first-repo>.extraheader "AUTHORIZATION: bearer $(System.AccessToken)"
    git config --global http.https://dev.azure.com/<your-org>/<your-second-repo-project>/_git/<your-second-repo>.extraheader "AUTHORIZATION: bearer $(System.AccessToken)"

You will need to set the extraheader for each of the called git repos.

Beware that you might need to unset the extraheader after your terraform calls if your pipeline sets the extraheader several times on the same worker. This is because git can get confused with multiple extraheader declaration. You do this by adding to following step:

- bash: |
    git config --global --unset-all http.https://dev.azure.com/<your-org>/<your-first-repo-project>/_git/<your-first-repo>.extraheader
    git config --global --unset-all http.https://dev.azure.com/<your-org>/<your-second-repo-project>/_git/<your-second-repo>.extraheader
-1
votes

I don't think you can. Usually, you create another build and link to the artifacts from that build to use it in your current definition. That way you don't need to connect to a different Git repository