How can I create a pipeline as code in AWS codepipeline

2
votes

I am using AWS codepipeline as my CI/CD tool. I have a code pipeline template yml file on my git and I wonder how I can link the file to AWS codepipeline. What I want to do is to let codepipeline to create/update the pipeline based on my pipeline yml file in github.

I have searched and tried on AWS console. All I can do is to manually create a pipeline via console and upload the template file. It works but it is not pipeline as code. If I want to change the stages in the pipeline, I will have to manually update the pipeline on AWS console or via cloudformation command.

Let me give an example, if I need to add a new stage in my pipeline. What I'd like to do is to update the yml file in github repo and commit it, then AWS codepipeline reads this yml file to update itself. I don't want to manually update the stage via AWS console.

Is there a way for me to sync the codepipeline to my pipeline yml file under source control?

1
amazon-web-servicescontinuous-integrationaws-codepipeline
You mean pipeline in a pipeline? For that you would have to have a master pipeline outside of your code. The master pipleline would construct a child pipeline based on your yml from the git repo. – Marcin
no, I don't mean that. I mean pipeline as code. – Joey Yi Zhao
You mean feature like GitHub actions? Where you keep yml pipeline in Source Code repo and same is fetched in next execution? – saurabh14292
what I mean is that I keep pipeline yml file in github. And let AWS pipeline read this yml file to update its pipeline. For example, if I want to add a new stage in AWS pipeline, I don't need to go to AWS console. All I need is to update the yml file in github. – Joey Yi Zhao
You can certainly create a codepipeline in cloudfromation. However, you won't be able to run a pipeline which updates itself, without first finishing the pipeline executions. Therefore, you will need to handle some sort of additional feedback loop for cloudformation states. Your better off creating a one of pipeline which is used to update your other pipelines. – pkarfs

1 Answers

2
votes

I have seen lot of people wondering about this setup where everything is managed via code and I personally use this too with CodePipeline. I can see many people have replied but let me put it here with detials so that it can be help to anyone who wants to do this.

There are two ways to achieve this and let me try to explain both option here:

Option:1

Create two Seperate Pipeline:

"Pipeline -1" (Responsible for config change like adding extra stages to main pipeline "Pipeline -2", with two stage source and deploy (CloudFormation)

source_Config (gitrepo_config) --> deploy_Config_Cfn

"Pipeline -2" (Actual deployment Pipeline with stages like source, buid, deploy stage which will be created by using resource.yaml)

source_Resource (gitrepo_resource) --> build_Resource --> Deploy_Resource

  1. Based on above config upload the template you use to create the main pipeline "resource.yaml" to repo "gitrepo_config".

  2. Upload all the Code in repo "gitrepo_resource" based on the deployment provide you are using for "Deploy_Resource"

Once above setup is done when you want to put extra stages in pipeline you can make changes in file "resource.yaml" in git repo and "Pipeline -1" will do the rest.

Option:2 (Little Complex But let me see if I can explain)

I was using option 1 until I came up with this option.

This second way is like 100% code because even in above option I have to create the "Pipeline -1" either manually or via CFN for first time and later for update also I need to go to console.

To overcome this we can include both Pipeline in same CloudFormation template "resource.yaml" and only one time we have to execute that CloudFormation stack and later everything else is automatic.

I hope this will be helpful to everyone.

Note: Also we have to keep in mind in both option if during any config change if pipeline execution is in progress for resource pipeline "Pipeline -2 " then it might be marked as failed so to overcome this issue you can always set additional trigger which will trigger the "Pipeline -2" based on success state of "Pipeline -1" in addition to the source code trigger.