Can't use localstack dynamoDB to lock terraform state: UnrecognizedClientException

Question

I've been trying to create a local development environment to play with terraform with localstack (https://github.com/localstack/localstack) running on docker.

I was already able to create a S3 bucket to store the terraform state, but I also wanted to simulate the DynamoDB as lock.

The configuration is:

localstack docker-compose.yml:

version: "3.2"
services:
  localstack:
    image: localstack/localstack:latest
    container_name: localstack
    ports:
      - "4563-4599:4563-4599"
      - "8080:8080"
    environment:
      - DATA_DIR=/tmp/localstack/data
      - DEBUG=1
    volumes:
      - "./.localstack:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

First terraform:

Used as initial bootstrap to create s3 tfstate storage and DynamoDB table for tfstate lock.

provider "aws" {
  region                      = "us-east-1"
  access_key                  = "foo"
  secret_key                  = "bar"
  skip_credentials_validation = true
  skip_requesting_account_id  = true
  skip_metadata_api_check     = true
  s3_force_path_style         = true
  endpoints {
    apigateway     = "http://localhost:4566"
    cloudformation = "http://localhost:4566"
    cloudwatch     = "http://localhost:4566"
    dynamodb       = "http://localhost:4566"
    es             = "http://localhost:4566"
    firehose       = "http://localhost:4566"
    iam            = "http://localhost:4566"
    kinesis        = "http://localhost:4566"
    lambda         = "http://localhost:4566"
    route53        = "http://localhost:4566"
    redshift       = "http://localhost:4566"
    s3             = "http://localhost:4566"
    secretsmanager = "http://localhost:4566"
    ses            = "http://localhost:4566"
    sns            = "http://localhost:4566"
    sqs            = "http://localhost:4566"
    ssm            = "http://localhost:4566"
    stepfunctions  = "http://localhost:4566"
    sts            = "http://localhost:4566"
  }
}

resource "aws_s3_bucket" "terraform_state" {
  bucket = "terraform-state"
  acl    = "private"

  versioning {
    enabled = true
  }

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state_access" {
  bucket = aws_s3_bucket.terraform_state.id

  block_public_acls       = true
  ignore_public_acls      = true
  block_public_policy     = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "terraform_state_lock" {
  name           = "terraformlock"
  read_capacity  = 5
  write_capacity = 5
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

Second terraform:

Creates resources and will store the state in s3 and use DynamoDB to create lock.

terraform {
  backend "s3" {
    bucket                      = "terraform-state"
    key                         = "main/terraform.tfstate"
    region                      = "us-east-1"
    endpoint                    = "http://localhost:4566"
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    force_path_style            = true
    dynamodb_table              = "terraformlock"
    encrypt                     = true
  }
}

provider "aws" {
  region                      = "us-east-1"
  access_key                  = "foo"
  secret_key                  = "bar"
  skip_credentials_validation = true
  skip_requesting_account_id  = true
  skip_metadata_api_check     = true
  s3_force_path_style         = true
  endpoints {
    apigateway     = "http://localhost:4566"
    cloudformation = "http://localhost:4566"
    cloudwatch     = "http://localhost:4566"
    dynamodb       = "http://localhost:4566"
    es             = "http://localhost:4566"
    ec2            = "http://localhost:4566"
    firehose       = "http://localhost:4566"
    iam            = "http://localhost:4566"
    kinesis        = "http://localhost:4566"
    lambda         = "http://localhost:4566"
    route53        = "http://localhost:4566"
    redshift       = "http://localhost:4566"
    s3             = "http://localhost:4566"
    secretsmanager = "http://localhost:4566"
    ses            = "http://localhost:4566"
    sns            = "http://localhost:4566"
    sqs            = "http://localhost:4566"
    ssm            = "http://localhost:4566"
    stepfunctions  = "http://localhost:4566"
    sts            = "http://localhost:4566"
  }
}

resource "aws_sqs_queue" "test" {
  name = "test"
  tags = {
    "Environment" = "dev"
  }
}

resource "aws_sns_topic" "test" {
  name         = "test"
  display_name = "test"
}

Whenever I apply the second terraform, I get this error:

❯ terraform apply
Acquiring state lock. This may take a few moments...

Error: Error locking state: Error acquiring the state lock: 2 errors occurred:
        * UnrecognizedClientException: The security token included in the request is invalid.
        status code: 400, request id: UEGJV0SQ614NIEDRB93IAF0JQ7VV4KQNSO5AEMVJF66Q9ASUAAJG
        * UnrecognizedClientException: The security token included in the request is invalid.
        status code: 400, request id: U1IRF6CHGK7RM4SQEGVCSU699RVV4KQNSO5AEMVJF66Q9ASUAAJG



Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and try
again. For most commands, you can disable locking with the "-lock=false"
flag, but this is not recommended.

Anyone have ever tried this or would have idea about what is causing it?

Marcin Marcin · Accepted Answer · 2021-01-25T00:00:36

This probably happens because you are trying to use real DynamoDB, rather then from localstack. To use localstack, you have to add

dynamodb_endpoint           = "http://localhost:4566"

to your backend.S3 configuration. Once you updated your backend setup, you will have to reinitialize your TF using terraform init.

Can't use localstack dynamoDB to lock terraform state: UnrecognizedClientException

1 Answers