Is it possible to to “variableize” the source path for a module in Terraform?

Question

Is it possible to use a variable for the module source in Terraform or Terragrunt. Using Terragrunt I know we can override the module source to a local directory but it does not seem to allow us to use a different repo.

The use case is to support a development repository and a live repository. Developers will use a different repository for development of modules than will be used for the production/live deployments.

I am familiar with using the Terragrunt approach to separate environments. We can go that route, e.g. the configurations in the live folders would point to one repo and the configurations in the dev/qa folders point to another repo.

Code Snippet:

module "s3_module" {
  source = "${var.source_url}"
  bucket_name = "thereoncewasakingguardinghisgardenallalone"
}

Error:

Error downloading modules: Error loading modules: error downloading 'file:///home/vagrant/code/Terraform/Examples/Lab-US-West-1/${var.source_url}': source path error: stat /home/vagrant/code/Terraform/Examples/Lab-US-West-1/${var.source_url}: no such file or directory

Martin Atkins Martin Atkins · Accepted Answer · 2019-09-17T17:28:34

Terraform does not allow varying module sources like this because module installation happens during terraform init and thus must make all of its decisions statically before evaluating the main code, similarly to how dependency installation works in many other languages.

A different way to meet your goal of giving the production automation a different view of modules than other callers (such as developers) is to use Terraform's native module registry mechanism and its associated service discovery protocol.

To do this requires running a service that implements the registry protocol, which is essentially just an extra level of indirection over module sources that allows them to be decided by the remote server rather than hard-coded in the configuration. If you have your module registry running at terraform.example.com then your module source strings might look something like this:

module "s3_module" {
  source = "terraform.example.com/any-namespace/s3/aws"

  bucket_name = "thereoncewasakingguardinghisgardenallalone"
}

The registry protocol can return module source addresses of any type that Terraform supports, including git:: for git repositories. Therefore you could set up the registry so that the above module address leads to a normal Git repository so that it's convenient for developers.

By default, Terraform will use its service discovery protocol to find the location of the registry API for terraform.example.com. You should set up the main service discovery document to refer to the registry that would be used outside of production, in order to avoid the need for manual configuration on each developer's system.

In your production system -- where presumably Terraform is running in some sort of automation -- you can use a CLI Configuration setting to override the discovery of terraform.example.com to point to a different registry API that is more appropriate for your production environment:

# Note that this goes in the _CLI configuration_, which is *not* the
# same thing as the .tf files that describe your infrastructure.

host "terraform.example.com" {
  services = {
    "modules.v1" = "https://production-terraform.example.com/modules/"
  }
}

With that CLI configuration in place, Terraform will interpret terraform.example.com differently and use this other registry API instead, where you can potentially arrange for it to select only packaged modules in an AWS S3 bucket, or whatever other constraint seems appropriate for production.

Terraform Cloud and Enterprise have a built-in private module registry, but you can deploy anything that talks the same protocol on your own infrastructure and use it instead if you set up the service discovery protocol correctly. HashiCorp doesn't have an official private registry you can run yourself, but there are some community implementations of the protocol and the most important parts of the protocol (listing available versions and finding a download URL) are simple enough that they can be backed by a static website served from AWS S3, or similar.

Is it possible to to “variableize” the source path for a module in Terraform?

1 Answers