AWS codeBuild/codePipeline with serverless framework

Question

I am trying to automate Deployment Pipeline for my application. Here is the automation architecture, I came up with:

As you can see, I am using codePipeline and codeBuild to automate my deployment. My backend is based on Serverless Framework, which deploys lambda functions on firing sls deploy command. This is the reason, I did not use codeDeploy to do traditional deployment. buildspec.yml file looks like this:

version: 0.1

phases:
  install:
    commands:
      – apt-get -y update
      – npm install -g [email protected]
  build:
    commands:
      – cd nj2jp/serverless && npm install
  post_build:
    commands:
      – serverless deploy –verbose

artifacts:
  files:
    – serverless.yml
  discard-paths: yes

Now, I have 3 Questions in regards to CodeBuild and Serverless:

Question 1: The command sls deploy depends on a file called config.yml which contains secrets such as db password. This file will not be checked into git. What do you think is the best possible way to include config.yml in codeBuild?

Question 2: Rollbacks can be done with AWS, if we have to deploy traditional EC2 applications using codeDeploy. In case of serverless, we are not using codeDeploy and serverless also supports rollback features. How do we leverage serverless rollback within the codePipeline?

Question 3: Triggering codePipeline when a Pull Request happens. I saw some posts saying, it is not supported by codePipeline. But those posts were from last year, Is Pull Request now supported by codePipeline?

Hack Answers(Not correct, but works. Need better answers from you.)

Answer 1: The config.yml file can be saved in a private S3 bucket and can be pulled to codeBuild as part of pre-build setup or We can add all secrets to codeBuild's Env variables. I dont like the second option, as I want to have consistent across all environments. Any better solutions for this problem?

Answer 2: I cannot think of an hack for this. Looking out for answers from you.

Answer 3: I came across some blog posts that use [APIGateway + Lambda + S3] to trigger codePipeline for pull requests. But I feel, this feature has to be provided as an out-of-box one. Is there any updates on the codePipeline for this feature?

Answer 1

Question 1

Update:

Serverless framework now supports referencing variables from the Parameter Store. That means you can skip defining them in CodeBuild as serverless will retrieve them from Parameter Store upon deploy.

Example:

serverless.yaml

provider:
  name: aws
  runtime: nodejs8.10
  region: us-west-2
  stage: ${env:REGION}
  environment:
    S3_BUCKET: ${env:/s3/bucket}
    # Use ~true for SecureString parameters
    DB_USERNAME: ${ssm:/db/username~true}
    DB_PASSWORD: ${env:/db/password~true}

---

Original Answer:

If you want to stick with your config.yml, then the only way to make it work is through hacks similar to what you are already doing since it is not version-controlled.

What I'd suggest is to have your environment variables stored in EC2 Parameter Store which you can reference in your CodeBuild buildspec.yml. These variables are accessible in your serverless.yml using ${env:VARIABLE_NAME}.

For local development, you should also use real environment variables versus storing them in config.yml. Tools such as direnv are great at this.

Question 2

You can do manual rollbacks by re-running the previous CodeBuild job. I can't think of an easy way to do it automatically like what CodeDeploy does. Maybe a Lambda function can do a post-deploy test and if it fails, it can trigger re-running the previous CodeBuild job.

Question 3

CodePipelines are tied to a single branch so to make it work on PR branches, you have to do hacks like the article you mentioned. I have resorted to using API Gateway + Lambda + CodeBuild (No CodePipeline) to do this.

Answer 2

Just to add to the accepted answer given for Question 1 (Thanks @dashmug and @Lakshman Diwaakar)

This does sort out how to get the Parameter Store values into your Lambda. However, the values are shown as plain text in the Lambda console. I need to work out next is how to add encryption.

Parameter Store

Add your env variables as parameters in AWS Systems Manager > Parameter store

https://eu-west-1.console.aws.amazon.com/systems-manager/parameters

buildspec.yml

version: 0.2    
phases:
  install:
    commands:
      - npm install
      - npm install -g serverless
  build:
    commands:
      - serverless deploy

serverless.yml

Reference the created Params in serverless.yml

Append ~true for secure strings.

provider:
  name: aws
  runtime: nodejs6.10
  region: eu-west-1
  stage: prod
  environment:
    FOO: ${ssm:/app/production/foo}
    DB_USERNAME: ${ssm:/app/production/myDatabase/username~true}
    DB_PASSWORD: ${ssm:/app/production/myDatabase/password~true}

handler.js

Use the env vars in your handler

export async function main (event, context, callback) {

  console.log('process.env.FOO', process.env.FOO)
  console.log('process.env.DB_USERNAME', process.env.DB_USERNAME)
  console.log('process.env.DB_PASSWORD', process.env.DB_PASSWORD)

  callback(null, ok('success'))
}