The project is for an insurance company. We’ve set up the infrastructure on AWS. Architecture is rather complex, using VPC, EC2, LBs, EKS, RDS, Lambdas etc. services all inter-connected.
Current setup:
- The entire infrastructure is described with CloudFormation templates
- Templates are stored and versioned on GitHub, each piece of infrastructure (usually one CloudFormation template) has its own git repository
- We have a set of bash scripts that we use to apply templates using CloudFormation CLI
- Bash scripts also use configuration files (configs for each environment), also stored in git in a separate repository, which are used as input parameters for different CFN templates
- Scripts are currently executed with Jenkins, with a separate job for each piece of infrastructure
The problem:
- Whenever I need to add a new piece of infrastructure I have to create a new git repo, update the config file, configure a new job in Jenkins
- Maintaining Jenkins is yet another problem
- Sharing and distributing among multiple teams and projects is hard, ending up with me being the one who initially sets up and maintains the infrastructure in all projects
- The CI/CD process is not easy to distribute within and out of my team, because of too many implementation details
- With the number of CFN templates, it’s becoming a mess
Recreating the infrastructure in one transaction would save a lot of time. For this, I was looking at using CloudFormation nested stacks. Each nested stack would be one piece of my infrastructure.
If you’ve had the same or similar problem, how did you managed to resolve it? What tools and structure are you currently using?