CDK pipeline for multiple branches at the same time

Question

I would like to have the CI-CD pipeline deployed for the develop and master branch at all times. Pipeline for feature branches to be created manually developers as and when needed.

I am using the pipeline from https://github.com/awslabs/aws-simple-cicd

In project-config.json , we have :

"Backend": [
  {
    "pipelineName": "backend",
    "ccRepoName": "backend",
    "branch": "master",
    "type": "BitBucket",
    "cron": ""
  }
],

Step 1. Pipeline deployed for branch master

Step 2. Edit project-config.json and change branch name:

 "Backend": [
      {
        "pipelineName": "backend",
        "ccRepoName": "backend",
        "branch": "develop",
        "type": "BitBucket",
        "cron": ""
      }
    ],

Step 3. Pipeline deployed for develop branch

At this stage, it deletes the pipeline for the master branch and deploys it for the develop branch. How can we keep the pipeline for multiple branches at the same time?

Answer 1

You can define multiple pipeline stacks based on which branch you'd like to build

I've successfully done this using the following method (using Python)

# app.py
from aws_cdk import core
from my_project.pipeline_stack import PipelineStack

app = core.App()

PipelineStack(app, "project-master-branch",
    branch="main"
)

PipelineStack(app, "project-develop-branch",
    branch="develop"
)

With the PipelineStack looking like

# my_project/pipeline_stack.py

from aws_cdk import (
    core,
    pipelines,
    aws_codepipeline as codepipeline,
    aws_codepipeline_actions as cpactions
)

class PipelineStack(core.Stack):
    def __init__(self, scope: core.Construct, id: str, branch: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        source_artifact = codepipeline.Artifact()
        cloud_assembly_artifact = codepipeline.Artifact()

        pipeline = pipelines.CdkPipeline(
            self,
            "Pipeline",
            cloud_assembly_artifact=cloud_assembly_artifact,
            pipeline_name=f"my-pipeline-{branch}",
            source_action=cpactions.BitBucketSourceAction(
                ... # your Bitbucket details
                branch=branch  # taking the main or develop branch
            ),
            synth_action=pipelines.SimpleSynthAction(
                source_artifact=source_artifact,
                cloud_assembly_artifact=cloud_assembly_artifact,
                install_command="npm i -g aws-cdk",
                synth_command="cdk synth",
                ... # other options you can provide
            )
        )

This will create two codepipelines that respond and deploy in response to changes to main and develop

Answer 2

The answer provided by maafk is a great answer. However in our case, for compliance reasons, we could not have different pipelines for various environments provisioned from the same AWS account. So, in our environment we have a development AWS account, a staging one and a production one. Each pipeline for dev, staging and production environments needs to be provisioned from it's own AWS account. Also, our production account needs to be fully isolated from the other accounts with no connectivity, to or from the dev and staging ones. On top of that, our developers need to create pipelines for different dev branches. There was no easy way for us to manage all this, without ending up creating all sort of pipelines across multiple aws accounts, where we only needed one for production and one for staging. We came up with the following solution which typically allows for any number of pipelines to be deployed, without the need to check in the cdk source code any branch related bindings.

We are creating a CodePipeline as below:

const branch = this.node.tryGetContext("branch");

new CodePipeline(this, createResourceName(this.node, "CodePipeline"), {
  pipelineName: createResourceName(this.node, "CodePipeline"),
  synth: new CodeBuildStep("Synth", {
    input: ...,
    env: { // The trick is to create an environment variable where the branch name is stored.
      "BRANCH": branch,
    },
    commands: [
      "npm ci",
      "npm run build",
      "npx cdk synth -c branch=$BRANCH", // During synthesis the environment variable is passed in as the branch name to the cdk context.
    ]
  })
});

// This is just for generating resource names based on the pipeline branch
export function createResourceName(node: ConstructNode, resourceName: string) {
  const branch = node.tryGetContext("branch");
  return `${branch}-${resourceName}`;
}

The trick here is in how the branch variable is being resolved during synthesis. It is actually being looked up from the cdk context. The first time a developer provisions a pipeline from a new dev branch, say for instance branch "bug123", the cdk application is deployed using a command like: cdk deploy -c branch=bug123. This is typically executed from a local workstation. During synthesis, the branch is retrieved from the cdk context. At the same time, the first time deployment creates an environment variable with the name BRANCH in the CodeBuild project responsible for the synthesizing the CDK app. This is like writing the branch name for the pipeline in stone.

The second time the pipeline runs, say for example because the developer checked in some code to the branch, the command that runs for synthesizing the app is the one defined in the commands section of the pipeline: npx cdk synth -c branch=$BRANCH. Since the environment variable was set during the first time deployment, now the same branch name is being passed to the cdk context. As a result the branch variable in the cdk code is again resolved from the context, to the original branch name the pipeline was created with.

This way, it is possible to create multiple branches and deploy new pipelines for each one, without worrying or maintaining any mapping between branches and pipelines in the source code.