lambda node.js function suddenly failing with signature error

Question

I have a lambda function that I've been using for well over a month to create thumbnails from uploaded images. I have it configured so it can trigger from an event or be invoked manually and/or via a gatewayAPI rest call. As mentioned, I have already been using this consistently and I am now building the local content that will use it.

It takes an image in one bucket, and creates a thumbnail in another bucket. To do this, I have a node.js async waterfall that follows the steps:

use s3client to getObject from first bucket
use 'sharp' to create a thumbnail
use 'sharp' to get image properties for metadata (width, height, type)
use s3client to putObject to thumbnail bucket

It all of sudden decided to fail on step 4 and I'm not sure why:

2019-11-20T16:36:49.656Z    8ab486b1-e676-4862-9dec-2264166e8c0f    INFO    putObject failed
 { SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided.
   Check your key and signing method.
    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/services/s3.js:585:35)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
  message:
   'The request signature we calculated does not match the signature you provided. Check your key and signing method.',
  code: 'SignatureDoesNotMatch',
  region: null,
  time: 2019-11-20T16:36:49.656Z,
  requestId: '25F4E6AE39104A62',
  extendedRequestId:
   'n8gPJ8y+bzhv7yHEiYDsNgjxyZqCENiRY0mUrZs/6Mp66DrN/CGBvWUcrKHNYF545k8zqKSMKVg=',
  cfId: undefined,
  statusCode: 403,
  retryable: false,
  retryDelay: 63.05576607189243 }

To my knowledge, I have not changed anything that should effect how it works (I haven't changed any permissions on the thumbnail bucket or for that matter the original bucket)

        async.waterfall([
                function download(next) {
                    citScope.getS3client().getObject({
                            Bucket: image-files,
                            Key: key
                        },
                        next)
                },
                function transform(response, next) {
                    metadata.size = response.ContentLength.toString()
                    metadata.mtime = Math.floor(AWS.util.date.unixTimestamp(response.LastModified)).toString()

                    sharpImage = sharp(response.Body)
                        .metadata(next)
                },
                function parseMetadata(md, next) {
                    metadata.imagetype = `image/${md.format}`
                    metadata.width = md.width.toString()
                    metadata.height = md.height.toString()
                    sharpImage.clone()
                        .resize(MAX_WIDTH, MAX_HEIGHT, {fit: sharp.fit.inside})
                        .jpeg()
                        .toBuffer(next)
                },
                function upload(data, info, next) {
                    citScope.getS3client().putObject({
                            "Bucket": image-thumbnails,
                            "Key": thumbKey,
                            "Body": data,
                            "ContentType": "image/jpeg",
                            "Metadata": metadata
                        },
                        next)
                }
            ], function (err) {
                if (err) {
                    let output = {
                        "message": `Unable to resize image-files/${key}`,
                        "error": err
                    }
                    citReject(output)
                } else {
                    let output = {
                        "success":1,
                        "message": `Successfully resized s3://image-files:${key}`,
                        "metadata": metadata
                    }
                    citResolve(output)
                }
            }
        )

I'm using - and have been using with no problems - a lambda execution role that at present has AmazonS3FullAccess added to it. Obviously, the getObject works fine. An earlier step does a getHeadObject to verify the file exists before starting the process of trying to create a thumbnail. It works fine. I have also confirmed that another function I have which does a 'putObject' of an empty file to create a pseudo-directory placeholder appears to work just fine (but on the image bucket, not the thumbnail bucket)

I've gone over the permissions and there doesn't seem to be anything out of whack between the two buckets, but I'm not entirely sure what to look for. And as stated, I am unaware of anything I might have changed to doink it up all of a sudden.

Any help in tracking down why this suddenly stopped working is appreciated.

Scott Scott · Accepted Answer · 2019-11-20T18:40:25

OMFG - lambda errors are useless.

After screwing with this for a day-and-a-half trying everything under the sun, I finally isolated the problem. I have more than one function that creates metadata to be added to new objects.

As I was creating another function to create similar metadata, I noticed that I had camelcase for imageType in one function and all-lower-case imagetype in another. So I changed the camel-case version to lower case. Or so I thought. I forgot that I had initialized the array in this function with the camelCase version as well so it was inserting an empty string for imageType that was never being replaced with a value. The value was being [properly] inserted under the imagetype key.

For whatever reason, amazon in it's infinite wisdom decided to throw a signature does not match error because the imageType = '' was in there. As soon as I corrected the case of that (which I just noticed as going through the code doing testing and didn't think it would actually do anything for the problem) it started working. Since I did that in passing, it even took me a few minutes to realized that was what actually fixed it.

smh

More info on why the metadata:

In the case of the thumbnails, I don't have control over all the ways an image can be uploaded (thus the mix of event trigger and manual trigger capability to create thumbnail images). As a result, I don't really feel like copying and renaming an image file to add metadata to it that our UI tools need such as the image type, pixel width and height. The metadata in the example above is instead extracted when the thumbnail is created as that information is available so I just tag it on the thumbnail itself. Then if I need to know the image's pixel width and pixel height, I can just pull the header for the image to get those details without needing to download the entire image and looking inside it. I don't even have to pull the entire thumbnail this way. The mtime and size are included for comparison to the actual image's header to make sure the thumbnail itself is current with the corresponding image in the other bucket.

lambda node.js function suddenly failing with signature error

1 Answers