CodePipeline, CodeBuild, CloudFormation, Lambda: build multiple lambdas in a single build and assign their code correctly

Question

I have a CD pipeline built with AWS CDK and CodePipeline. It compiles the code for 5 lambdas, each of which it pushes to a secondary artifact.

The S3 locations of each of the artifacts are assigned to the parameters of a CloudFormation template which are attached to the Code parts of the lambdas.

This is working fine!

My problem is, I cannot add a sixth secondary artifact to CodeBuild (hard limit). I also cannot combine all of my lambda code into a single artifact as (as far as I can see) CodePipeline is not smart enough to look inside an artifact when assigning Code to a lambda in CloudFormation.

What is the recommendation for deploying multiple lambdas from a CodeBuild/CodePipeline? How have other people solved this issue?

EDIT: Code pipeline CF template note: I have only included 2 lambdas as an example

Lambda application template

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "Lambda1": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Code": {
          "S3Bucket": {
            "Ref": "lambda1SourceBucketNameParameter3EE73025"
          },
          "S3Key": {
            "Ref": "lambda1SourceObjectKeyParameter326E8288"
          }
        }
      }
    },
    "Lambda2": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Code": {
          "S3Bucket": {
            "Ref": "lambda2SourceBucketNameParameter3EE73025"
          },
          "S3Key": {
            "Ref": "lambda2SourceObjectKeyParameter326E8288"
          }
        }
      }
    }
  },
  "Parameters": {
    "lambda1SourceBucketNameParameter3EE73025": {
      "Type": "String"
    },
    "lambda1SourceObjectKeyParameter326E8288": {
      "Type": "String"
    },
    "lambda2SourceBucketNameParameterA0D2319B": {
      "Type": "String"
    },
    "lambda2SourceObjectKeyParameterF3B3F2C2": {
      "Type": "String"
    }
  }
}

Code Pipeline template:

{
  "Resources": {
    "Pipeline40CE5EDC": {
      "Type": "AWS::CodePipeline::Pipeline",
      "Properties": {
        "Stages": [
          {
            "Actions": [
              {
                "ActionTypeId": {
                  "Provider": "CodeBuild"
                },
                "Name": "CDK_Build",
                "OutputArtifacts": [
                  {
                    "Name": "CdkbuildOutput"
                  }
                ],
                "RunOrder": 1
              },
              {
                "ActionTypeId": {
                  "Provider": "CodeBuild"
                },
                "Name": "Lambda_Build",
                "OutputArtifacts": [
                  { "Name": "CompiledLambdaCode1" },
                  { "Name": "CompiledLambdaCode2" }
                ],
                "RunOrder": 1
              }
            ],
            "Name": "Build"
          },
          {
            "Actions": [
              {
                "ActionTypeId": {
                  "Provider": "CloudFormation"
                },
                "Configuration": {
                  "StackName": "DeployLambdas",
                  "ParameterOverrides": "{\"lambda2SourceBucketNameParameterA0D2319B\":{\"Fn::GetArtifactAtt\":[\"CompiledLambdaCode1\",\"BucketName\"]},\"lambda2SourceObjectKeyParameterF3B3F2C2\":{\"Fn::GetArtifactAtt\":[\"CompiledLambdaCode1\",\"ObjectKey\"]},\"lambda1SourceBucketNameParameter3EE73025\":{\"Fn::GetArtifactAtt\":[\"CompiledLambdaCode2\",\"BucketName\"]},\"lambda1SourceObjectKeyParameter326E8288\":{\"Fn::GetArtifactAtt\":[\"CompiledLambdaCode2\",\"ObjectKey\"]}}",
                  "ActionMode": "CREATE_UPDATE",
                  "TemplatePath": "CdkbuildOutput::CFTemplate.template.json"
                },
                "InputArtifacts": [
                  { "Name": "CompiledLambdaCode1" },
                  { "Name": "CompiledLambdaCode2" },
                  { "Name": "CdkbuildOutput" }
                ],
                "Name": "Deploy",
                "RunOrder": 1
              }
            ],
            "Name": "Deploy"
          }
        ],
        "ArtifactStore": {
          "EncryptionKey": "the key",
          "Location": "the bucket",
          "Type": "S3"
        },
        "Name": "Pipeline"
      }
    }
  }
}

Are they in different buckets? Can't you put them all in one artifact store? — Marcin
Same bucket, same artifact store. I don't see how that relates to the issue, can you explain? — Shane
So where are those 5 artifacts of yours comming from that you hit the limit? — Marcin
Can you clarify your design. You have: CP source action -> 1 CB build action creating 5 CFN lambda templates -> 5 deploy actions with CFN providers which creates the lambdas from the templates (one per template)? — Marcin

Tim Bassett Tim Bassett · Accepted Answer · 2020-06-05T14:10:13

Reviewed templates.

So, I don't see five inputs to a CodeBuild action, but I do see 2 inputs to a CloudFormation action (Deploy).

I assume your problem was a perceived limit of 5 input to the CloudFormation action. Is that assumption correct?

The limits for a CloudFormation action are actually 10. See "Action Type Constraints for Artifacts " @ https://docs.aws.amazon.com/codepipeline/latest/userguide/reference-pipeline-structure.html#reference-action-artifacts

So if you can use up to 10, will that suffice?

If not, I have other ideas that would take a lot longer to document.

CodePipeline, CodeBuild, CloudFormation, Lambda: build multiple lambdas in a single build and assign their code correctly

1 Answers