Gitlab CI/CD with Terraform and 3rd party plugin

Question

I'm provisioning resources on AWS as well as on a 3rd party system via a custom provider. On my local environment, it works fine (as I have downloaded and compiled the 3rd party provider).

As a next step, I’m trying to run a Gitlab pipeline with Terraform.The AWS part works because it's a Terrafrom registered provider, however the 3rd party part fails (because it's not a registered Terraform provider).

When I try to run a Gitlab pipeline, it complains with an error and advises the following

In the latter case, the plugin must be installed manually by locating and downloading a suitable distribution package and placing the plugin’s executable file in the following directory: terraform.d/plugins/linux_amd64

The .gitlab-ci.yml file I have is the following:

image:
  name: hashicorp/terraform:light
  entrypoint:
    - '/usr/bin/env'
    - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

before_script:
  - rm -rf .terraform
  - terraform --version
  - export AWS_ACCESS_KEY
  - export AWS_SECRET_KEY
  - terraform init

stages:
  - validate
  - plan
  - apply

validate:
  stage: validate
  script:
    - terraform validate

plan:
  stage: plan
  script:
    - terraform plan -out "planfile"
  dependencies:
    - validate
  artifacts:
    paths:
      - planfile

apply:
  stage: apply
  script:
    - terraform apply -input=false "planfile"
  dependencies:
    - plan
  when: manual

I see an image is used called 'hashicorp/terraform:light'. Should I create a docker image based on this image with the 3rd party provider already in?

EDIT1

It's clear a custom image is required. I used the idea from rflume below and used the following Dockerfile to build the image.

# Multi-Stage builds require Docker Engine 17.05 or higher

# Build ACI provider
FROM ubuntu:bionic-20190515 as builder

ENV HOME /root
ENV GOPATH $HOME/go
ENV GOBIN $GOPATH/bin

RUN apt update &&\
    apt install -yqq software-properties-common \
    git \
    wget \
    unzip \
    gcc \
    perl \
    go-dep \
    make \
    build-essential &&\
    add-apt-repository ppa:longsleep/golang-backports &&\
    apt-get update &&\
    apt-get install -y golang-go &&\
    mkdir -p $GOPATH/src/github.com/terraform-providers &&\
    wget -O $HOME/terraform-provider-aci.zip https://github.com/ciscoecosystem/terraform-provider-aci/archive/master.zip &&\
    cd $GOPATH/src/github.com/terraform-providers/ &&\
    unzip $HOME/terraform-provider-aci.zip -d . &&\
    mv terraform-provider-aci-master \
    terraform-provider-aci-v1.0.0

WORKDIR $GOPATH/src/github.com/terraform-providers/terraform-provider-aci-v1.0.0

RUN dep ensure
RUN make build


# Build the actual image
FROM hashicorp/terraform:0.11.14


ENV GOBIN /root/go/bin
ENV PATH $GOBIN:$PATH

RUN mkdir -p /root/.terraform.d/plugins

COPY --from=builder /root/go/bin/terraform-provider-aci-v1.0.0 $GOBIN/terraform-provider-aci-v1.0.0

ENTRYPOINT ["/bin/sh", "-c"]

rflume rflume · Accepted Answer · 2019-06-11T18:38:24

The terraform:light image does not contain 3rd party providers. You can try using terraform:full image but you might have no luck with that one either.

In this case you'd have to build your own image based on one of the above images (preferably light, the smaller the image the better...).

I've had the same problem and built an image based on light with the AWS provider and an 3rd party Ansible provisioner Terraform-with-AWS-Provisioner-and-Ansible-Provider-for-Gitlab-CI which you can use as reference if you like.

EDIT1 (regarding your edit):

What I noticed is that you don't specify a provider version in your COPY --from=builder ... command. According to Plugin Names and Versions, you need to specify a version for the provider with _v[VERSION] to your built provider package though. Try that and let me know if it works then ;)

Gitlab CI/CD with Terraform and 3rd party plugin

EDIT1

2 Answers