CI/CD pipeline for AWS Glue

Question

I am fairly a new user of AWS Glue, which is one of new AWS managed services to orchestrate batch job workflows with ease.

• I have 3 different AWS IAM account (Dev, Test, Prod). Separate login for each account.
• Glue, Scala Jobs are version controlled in my Github repository.
• I want to build a CI/CD pipeline to automate the Test, Build and deployment of my Glue job as I commit and push in my Github repository.

I went through number of blogs and article describing CI/CD best practices. I found a good article (specifically on CI/CD in Data Pipeline workflows) on AWS Blog here. It is quite simple and has details of building CodePipeline using CodeCommit & AWS Cloudformation. But all the phases of CI/CD initiate and ends in the same AWS IAM account.

I have two questions:

1. Is separating Dev, Test and Prod account (different IAM) for building and managing Data Pipelines/ Data warehouse, a good practice?
2. If yes, how would I design CI/CD pipeline using 3 different IAM account?

Any suggestion on best practices to design CI/CD pipeline for AWS-Glue ?

Answer 1

Please refer to this which explains in detail on setting up a CI/CD pipeline across multiple accounts in a secured manner.

Is separating Dev, Test and Prod account (different IAM) for building and managing Data Pipelines/ Data warehouse, a good practice?

Yes it is a good practice.

If yes, how would I design CI/CD pipeline using 3 different IAM account?

Please refer to the shared link

Answer 2

Is separating Dev, Test and Prod account (different IAM) for building and managing Data Pipelines/ Data warehouse, a good practice?

Yes, separating environments in different AWS accounts is a good practice, it's recommended security practice as well.

If yes, how would I design CI/CD pipeline using 3 different IAM account?

Here is a blog which defines how you can configure resources in different AWS accounts and securely communicate with them.