UnrecognizedClientException when invoking a lambda from another lambda using localstack

Question

I am running localstack inside of a docker container with this docker-compose.yml file.

version: '2.1'

services:
  localstack:
    image: localstack/localstack
    ports:
      - "4567-4597:4567-4597"
      - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}"
    environment:
      - SERVICES=${SERVICES- }
      - DEBUG=1
      - DATA_DIR=${DATA_DIR- }
      - PORT_WEB_UI=${PORT_WEB_UI- }
      - LAMBDA_EXECUTOR=docker
      - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

To start localstack I run TMPDIR=/private$TMPDIR docker-compose up.

I have created two lambdas. When I run aws lambda list-functions --endpoint-url http://localhost:4574 --region=us-east-1 this is the output.

{
"Functions": [
    {
        "TracingConfig": {
            "Mode": "PassThrough"
        },
        "Version": "$LATEST",
        "CodeSha256": "qmDiumefhM0UutYv32By67cj24P/NuHIhKHgouPkDBs=",
        "FunctionName": "handler",
        "LastModified": "2019-08-08T17:56:58.277+0000",
        "RevisionId": "ffea379b-4913-420b-9444-f1e5d51b5908",
        "CodeSize": 5640253,
        "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:handler",
        "Environment": {
            "Variables": {
                "DB_NAME": "somedbname",
                "IS_PRODUCTION": "FALSE",
                "SERVER": "xxx.xxx.xx.xxx",
                "DB_PASS": "somepass",
                "DB_USER": "someuser",
                "PORT": "someport"
            }
        },
        "Handler": "handler",
        "Role": "r1",
        "Timeout": 3,
        "Runtime": "go1.x",
        "Description": ""
    },
    {
        "TracingConfig": {
            "Mode": "PassThrough"
        },
        "Version": "$LATEST",
        "CodeSha256": "wbT8YzTsYW4sIOAXLtjprrveq5NBMVUaa2srNvwLxM8=",
        "FunctionName": "paymentenginerouter",
        "LastModified": "2019-08-08T18:00:28.923+0000",
        "RevisionId": "bd79cb2e-6531-4987-bdfc-25a5d87e93f4",
        "CodeSize": 6602279,
        "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:paymentenginerouter",
        "Environment": {
            "Variables": {
                "DB_QUERY_LAMBDA": "handler",
                "AWS_REGION": "us-east-1"
            }
        },
        "Handler": "handler",
        "Role": "r1",
        "Timeout": 3,
        "Runtime": "go1.x",
        "Description": ""
    }
]

}

Inside the paymentenginerouter code I am attempting to call the handler lambda via:

    lambdaParams := &invoke.InvokeInput{
    FunctionName:   aws.String(os.Getenv("DB_QUERY_LAMBDA")),
    InvocationType: aws.String("RequestResponse"),
    LogType:        aws.String("Tail"),
    Payload:        payload,
}
result, err := svc.Invoke(lambdaParams)
if err != nil {
    resp.StatusCode = 500
    log.Fatal("Error while invoking lambda:\n", err.Error())
}

Where invoke is an import: invoke "github.com/aws/aws-sdk-go/service/lambda"

When I run the paymentenginerouter lambda via:

aws lambda invoke --function paymentenginerouter --payload  '{ "body": "{\"id\":\"12\",\"internalZoneCode\":\"xxxxxx\",\"vehicleId\":\"xxxxx\",\"vehicleVrn\":\"vehicleVrn\",\"vehicleVrnState\":\"vehicleVrnState\",\"durationInMinutes\":\"120\",\"verification\":{\"Token\":null,\"lpn\":null,\"ZoneCode\":null,\"IsExtension\":null,\"ParkingActionId\":null},\"selectedBillingMethodId\":\"xxxxxx\",\"startTimeLocal\":\"2019-07-29T11:36:47\",\"stopTimeLocal\":\"2019-07-29T13:36:47\",\"vehicleVin\":null,\"orderId\":\"1\",\"parkingActionType\":\"OnDemand\",\"digitalPayPaymentInfo\":{\"Provider\":\"<string>\",\"ChasePayData\":{\"ConsumerIP\":\"xxxx\",\"DigitalSessionID\":\"xxxx\",\"TransactionReferenceKey\":\"xxxx\"}}}"
}' --endpoint-url=http://localhost:4574 --region=us-east-1 out --debug

I receive this error:

localstack_1  | 2019/08/08 20:02:28 Error while invoking lambda:
localstack_1  | UnrecognizedClientException: The security token included in the request is invalid.
localstack_1  |     status code: 403, request id: bd4e3c15-47ae-44a2-ad6a-376d78d8fd92

Note

I can run the handler lambda without error by calling it directly through the cli:

aws lambda invoke --function handler --payload '{"body": "SELECT TOKEN, NAME, CREATED, ENABLED, TIMESTAMP FROM dbo.PAYMENT_TOKEN WHERE BILLING_METHOD_ID='xxxxxxx'"}' --endpoint-url=http://localhost:4574 --region=us-east-1 out --debug

I thought the AWS credentials are setup according the environment variables in localstack but I could be mistaken. Any idea how to get past this problem?

I am quite new to AWS lambdas and an absolute noob when it comes to localstack so please ask for more details if you need them. It's possible I am missing a critical piece of information in my description.

Ahmed Hussein Ahmed Hussein · Accepted Answer · 2019-11-06T10:02:10

The error you receiving The security token included in the request is invalid. means that your lambda is trying to call out to the real AWS with invalid credentials, rather than going to Localstack.

When running a lambda inside localstack and where the lambda code itself has to call out to any AWS services hosted in localstack, you will need to ensure that any endpoints used are re-directed to localstack.

To do this there is a documented feature in localstack found here:

https://github.com/localstack/localstack#configurations

LOCALSTACK_HOSTNAME: Name of the host where LocalStack services are available. This is needed in order to access the services from within your Lambda functions (e.g., to store an item to DynamoDB or S3 from Lambda). The variable LOCALSTACK_HOSTNAME is available for both, local Lambda execution (LAMBDA_EXECUTOR=local) and execution inside separate Docker containers (LAMBDA_EXECUTOR=docker).

Within your lambda code, ensure you use this environment variable to set the hostname (preceded with http:// and suffixed with the port number of that service in localstack, e.g. :4569 for dynamodb). This will ensure that the calls go to the right place.

Example in Go code snippet that would be added to your lambda where you are making a call to DynamoDB:

awsConfig.WithEndpoint("http://" + os.Getenv("LOCALSTACK_HOSTNAME") + ":4569")

UnrecognizedClientException when invoking a lambda from another lambda using localstack

1 Answers