2
votes

We have pipelines that pull the code from a CodeCommit repository and builds and deploys the code.

Between the Source stage and the Build stage, there is a manual approval stage and for random reason this stage sometimes gets skipped and the Pipeline continues directly to the Build stage without getting approved or rejected. Some other times even when it's approved, the manual approval stage would remain pending and waiting for approval. Some other times the Approval stage gets triggered simultaneously with the Source as soon as any code is pushed.

Weirdly enough this only happens to two pipelines that were created using a CloudFormation Template.

In a year or working with aws CodePipeline I never experienced such a thing.

Example of Approval stage getting skipped

Example of Approval stage getting skipped

Example of Approval stage getting triggered at the same time as the Source stage: Example of Approval stage getting triggered at the same time as the Source stage

The CloudFormation Template:

AWSTemplateFormatVersion: 2010-09-09
Description: >-
  Pipeline for React mobile WebApps. Creates a CodePipeline, along with a Deployment S3 that hosts the static and a CodeBuild Project to build and package the project 

Parameters:
  RepositoryName: 
    Type: String
    Description: Name of repository to build from
  RepositoryBranch:
    Type: String
    Description: The branch to pull from and build
    Default: master
    AllowedValues:
      - master
      - staging
  ApiURL:
    Type: String
    Description: domain of the api to be used by the web app
  SocketURL:
    Type: String
    Description: url of the socket to be used by the web app
  SentryURL:
    Type: String
    Description: url of sentry 
  CloudfrontURL:
    Type: String
    Description: url of storage

Resources:
  CodeBuildProject:
    Type: AWS::CodeBuild::Project
    DependsOn: CodeBuildRole
    Properties:
      Name: !Join
        - '-'
        - - !Ref RepositoryName
          - !Ref RepositoryBranch
          - build
          - project
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Environment: 
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/nodejs:8.11.0
        EnvironmentVariables:
        - Name: S3_URL
          Value: !Join
            - '-'
            - - !Ref RepositoryName
              - !Ref RepositoryBranch
        - Name: ENV
          Value: !Ref RepositoryBranch
        - Name: API_DOMAIN
          Value: !Ref ApiURL
        - Name: SOCKET_URL
          Value: !Ref SocketURL
        - Name: SENTRY_URL
          Value: !Ref SentryURL
        - Name: AWS_CLOUDFRONT_URL
          Value: !Ref CloudfrontURL
      Source:
        Type: CODEPIPELINE
        BuildSpec: !Sub |
          version: 0.2
          phases:
            install:
              commands:
                - echo "installing dependencies"
                - npm install
            pre_build:
               commands:
                - echo "building static files"
                - npm run build
                - echo "static files finished building"
            build:
              commands:
                - echo "build phase started"
                - aws s3 sync ./dist s3://${DeploymentBucket}/ --cache-control max-age=0
                - echo "build complete"
      Artifacts:
        Type: CODEPIPELINE
  CodeBuildRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - codebuild.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: codebuild-service
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action: "*"
            Resource: "*"
          Version: '2012-10-17'
  CodePipeline:
    Type: 'AWS::CodePipeline::Pipeline'
    DependsOn:
      - CodePipelineTrustRole
    Properties:
      Stages:
        - Name: Source
          Actions:
            - InputArtifacts: []
              Name: Source
              ActionTypeId:
                Category: Source
                Owner: AWS
                Version: '1'
                Provider: CodeCommit
              OutputArtifacts:
                - Name: MyApp
              Configuration:
                PollForSourceChanges: false
                BranchName: !Ref RepositoryBranch
                RepositoryName: !Ref RepositoryName
              RunOrder: 1
        - Name: Approval
          Actions:
            - InputArtifacts: []
              Name: Approval
              ActionTypeId: 
                Category: Approval
                Owner: AWS
                Version: '1'
                Provider: Manual
              OutputArtifacts: []
              Configuration:
              RunOrder: 1
        - Name: BuildAndDeploy
          Actions:
            - InputArtifacts:
                - Name: MyApp
              Name: CodeBuild
              ActionTypeId:
                Category: Build
                Owner: AWS
                Version: '1'
                Provider: CodeBuild
              OutputArtifacts:
                - Name: MyAppBuild
              Configuration:
                ProjectName: !Ref CodeBuildProject
              RunOrder: 1
        - Name: Invalidation
          Actions:
            - InputArtifacts: []
              Name: Invalidate-CloudFront
              ActionTypeId: 
                Category: Invoke
                Owner: AWS
                Version: '1'
                Provider: Lambda
              Configuration:
                FunctionName: "Cloudfront-Invalidator"
                UserParameters: !Sub '{"S3Bucket": "${DeploymentBucket}"}'
              OutputArtifacts: []
              RunOrder: 1
      ArtifactStore:
        Type: S3
        Location: pipeline-store-bucket
      RoleArn: !GetAtt
        - CodePipelineTrustRole
        - Arn
      Name: !Join 
        - '-'
        - - !Ref RepositoryName
          - !Ref RepositoryBranch
          - pipeline
  CodePipelineTrustRole:
    Type: 'AWS::IAM::Role'
    Description: Creates service role in IAM for AWS CodePipeline
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: 'sts:AssumeRole'
            Effect: Allow
            Principal:
              Service:
                - codepipeline.amazonaws.com
            Sid: 1
      Path: /
      Policies:
        - PolicyDocument:
            Statement:
              - Action:
                  - 's3:GetObject'
                  - 's3:GetObjectVersion'
                  - 's3:GetBucketVersioning'
                  - 's3:PutObject'
                Effect: Allow
                Resource:
                  - !Join
                    - ''
                    - - 'arn:aws:s3:::'
                      - 'pipeline-store-bucket'
                      - '/*'
                  - !Join
                    - ''
                    - - 'arn:aws:s3:::'
                      - 'pipeline-store-bucket'
              - Action:
                  - 'codecommit:CancelUploadArchive'
                  - 'codecommit:GetBranch'
                  - 'codecommit:GetCommit'
                  - 'codecommit:GetUploadArchiveStatus'
                  - 'codecommit:UploadArchive'
                Effect: Allow
                Resource:
                  - !Join 
                    - ':'
                    - - arn
                      - aws
                      - codecommit
                      - !Ref 'AWS::Region'
                      - !Ref 'AWS::AccountId'
                      - !Ref RepositoryName
              - Action:
                  - 'codebuild:StartBuild'
                  - 'codebuild:BatchGetBuilds'
                  - 'codebuild:StopBuild'
                Effect: Allow
                Resource: '*'
              - Action:
                  - 'cloudformation:DescribeStacks'
                  - 'cloudformation:DescribeChangeSet'
                  - 'cloudformation:CreateChangeSet'
                  - 'cloudformation:DeleteChangeSet'
                  - 'cloudformation:ExecuteChangeSet'
                Effect: Allow
                Resource: '*'
              - Action: 
                  - 'sns:Publish'
                Effect: Allow
                Resource: '*'
              - Action:
                  - 'lambda:*'
                  - 'cloudwatch:*'
                  - 'events:*'
                  - 'codepipeline:PutJobSuccessResult'
                  - 'codepipeline:PutJobFailureResult'
                Effect: Allow
                Resource: '*'
          PolicyName: CodePipelineTrustPolicy
      RoleName: !Join 
        - '-'
        - - !Ref AWS::StackName
          - CodePipeline
          - Role
  SourceEvent:
    Type: 'AWS::Events::Rule'
    Properties:
      Description: >-
        Rule for Amazon CloudWatch Events to detect changes to the source
        repository and trigger pipeline execution
      EventPattern:
        detail:
          event:
            - referenceCreated
            - referenceUpdated
          referenceName:
            - !Ref RepositoryBranch
          referenceType:
            - branch
        detail-type:
          - CodeCommit Repository State Change
        resources: 
            - !Join 
              - ':'
              - - arn
                - aws
                - codecommit
                - !Ref 'AWS::Region'
                - !Ref 'AWS::AccountId'
                - !Ref RepositoryName
        source:
          - aws.codecommit
      Name: !Join 
        - '-'
        - - !Ref RepositoryName
          - !Ref RepositoryBranch
          - SourceEvent
      State: ENABLED
      Targets:
        - Arn: !Join 
            - ':'
            - - arn
              - aws
              - codepipeline
              - !Ref 'AWS::Region'
              - !Ref 'AWS::AccountId'
              - !Join 
                - '-'
                - - !Ref RepositoryName
                  - !Ref RepositoryBranch
                  - pipeline
          Id: ProjectPipelineTarget
          RoleArn: !GetAtt SourceEventRole.Arn
  SourceEventRole:
    Type: 'AWS::IAM::Role'
    Description: >-
      IAM role to allow Amazon CloudWatch Events to trigger AWS CodePipeline
      execution
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: 'sts:AssumeRole'
            Effect: Allow
            Principal:
              Service:
                - events.amazonaws.com
            Sid: 1
      Policies:
        - PolicyDocument:
            Statement:
              - Action:
                  - 'codepipeline:StartPipelineExecution'
                Effect: Allow
                Resource:
                  - !Join 
                    - ':'
                    - - arn
                      - aws
                      - codepipeline
                      - !Ref 'AWS::Region'
                      - !Ref 'AWS::AccountId'
                      - !Join 
                        - '-'
                        - - !Ref RepositoryName
                          - !Ref RepositoryBranch
                          - pipeline
          PolicyName: CodeStarWorkerCloudWatchEventPolicy
      RoleName: !Join
        - '-'
        - - CodePipeline
          - !Ref RepositoryName
          - !Ref RepositoryBranch
          - CloudWatchEventRule
  DeploymentBucket:
    Type: 'AWS::S3::Bucket'
    Description: >-
      S3 Bucket to host the website built from the CodeCommit repository
    Properties:
      AccessControl: PublicRead
      WebsiteConfiguration:
        IndexDocument: index.html
        ErrorDocument: index.html
      BucketName: !Join 
        - '-'
        - - !Ref 'RepositoryName'
          - !Ref 'RepositoryBranch'
    DeletionPolicy: Delete
  DeploymentBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Description: >-
      Policy for the web hosting deployment S3 bucket
    Properties:
      Bucket: !Ref DeploymentBucket
      PolicyDocument:
        Statement: 
          - Sid: PublicReadForGetBucketObjectsxw
            Effect: Allow
            Principal: '*'
            Action: s3:GetObject
            Resource: !Join ['', ['arn:aws:s3:::', !Ref 'DeploymentBucket', /*]]
  CloudFrontDistribution:
    Type: AWS::CloudFront::Distribution
    Description: A CloudFront Distribution for the website hosting S3 buckets 
    DependsOn: DeploymentBucket
    Properties:
      DistributionConfig:
        Origins:
        - DomainName: !Join
            - '.'
            - - !Ref DeploymentBucket
              - s3
              - amazonaws
              - com
          Id: !Join
            - '-'
            - - S3
              - !Ref DeploymentBucket
          CustomOriginConfig:
            HTTPPort: 80
            HTTPSPort: 443
            OriginProtocolPolicy: https-only
        Enabled: true
        DefaultCacheBehavior:
          AllowedMethods:
            - GET
            - HEAD
            - OPTIONS
            - PUT
            - POST
            - PATCH
            - DELETE
          ForwardedValues:
            QueryString: 'false'
            Cookies:
              Forward: none
          TargetOriginId: !Join
            - '-'
            - - S3
              - !Ref DeploymentBucket
          ViewerProtocolPolicy: redirect-to-https
        IPV6Enabled: true
        DefaultRootObject: index.html

Outputs:
  PipelineURL:
    Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${CodePipeline}
    Description: URL for the CodePipeline of this stack
  SiteUrl:
    Value: !GetAtt [DeploymentBucket, WebsiteURL]
    Description: URL for the S3 Website
  CodeBuildUrl:
    Value: !Sub https://eu-west-1.console.aws.amazon.com/codebuild/home?${AWS::Region}#/projects/${CodeBuildProject}/view
    Description: URL for the CodeBuild Project of this stack
  RepositoryUrl:
    Value: !Sub https://eu-west-1.console.aws.amazon.com/codesuite/codecommit/repositories/${RepositoryName}/browse?region=eu-west-1
    Description: URL for the repository of this stack
1
can you also provide the CloudFormation template for these pipelines? We are also doing manual approval to pipelines that are created via cloudformation and it's working well as expected.groobie newbie
I added the CF templateQais Abou Jaoudé
That's weird, those approval stages looks alright aside from the empty Configuration: property, that isn't needed in the approval stage.groobie newbie

1 Answers

1
votes

Each CodePipeline stage can have a different version in it so that you can eg. test a new version in beta in parallel with the previous version being deployed to prod. All actions within a stage run the same set of versions (artifacts) though.

CodePipeline will not de-duplicate based on source commit ID because often it's desirable to re-release the same source commit (eg. because the build will pull updated dependencies, or because you want to reset things to a clean state after eg. a rollback).

This means there can be different versions with the same source commit ID in different stages. You go to the "View history" page to verify this.

Example of Approval stage getting skipped

In this print screen I suspect there's 2 changes. C1 was previously approved and is now deployed in "BuildAndDeploy". C2 is the change showing Succeeded in "Source" and waiting for approval in "Approval".

Example of Approval stage getting triggered at the same time as the Source stage:

This is also a case of 2 version. There's one version in Source and another in Approval.