Creating pipelines for Infrastructure as code seems easy to build task, but in a mature system, things can get complicated as it needs to handle many changing dynamics parts. A mature workflow for IAC not only automates the deployment of the IAC resources but also incorporates engineering fundamentals, resources validation, dependency management, test execution, security scanning, and more.
Symphony offers multiple workflows to ensure engineering excellence at every stage of the IaC process. The workflows are designed to be orchestrator independent by relying on the logic implemented in bash scripts and the pipelines being thin shim layers that execute the logic scripts. This design makes it easy to update the workflows or port on new tools regardless of the SCM by adding those tools to the workflow scripts.
This is the CI workflow to deploy the IaC resources to the target environment and subscription. It ensures best practices in IAC code repos, through multi-stage tasks to vet, deploy, test, all needed resources, and report any failures. Note that the details of stage execution may vary based on features available on the orchestrator's IAC tool.
This workflow has a set of inputs variable that can be updated at the execution time. While the inputs could vary based on the selected IaC tool, one common input variable across all is the environment name, which defines what environment configurations to be used.
This stage ensures code readiness. It executes pre_validate events, runs validations and linting tools, scans code for possible cred leaks, and executes any unit tests. Stage steps are executed in the following sequential order.
flowchart LR
A(Prep Env) --> B(Run Custom Scanners) --> C(Run IAC lint cmd)
C -->D(Run IAC validate cmd) --> E(Run IAC unit test)
E -->F(Finalize/Publish reports)
This stage plans the execution of the IAC code and estimates the scope of the changes. It initializes the IAC tool selected, runs plan/what-if commands to detect the changing scope, executes pre_deploy events, then runs deploy commands to update the resources, executes post_deploy events, and eventually ensures successful resource updates.
flowchart LR
A(Init IAC tool) --> B(Run IAC cmds to preview changes) --> C(Check for resources destroy operations)
C-->D(Run IAC Deploy cmds) -->E(Finalize/Publish reports)
This stage executes the integration or end-to-end tests against the recently deployed/updated resources to ensure the configurations/changes are reflected and resources are working as expected. It then publishes the results of the tests and drops them as artifacts for future reference.
flowchart LR
A(Init test framework) --> B(Execute e2e tests) -->E(Finalize/Publish reports)
This stage generates the needed scripts to repro the deployments, publish the created reports, and backup state files if required.
flowchart LR
A(Generate deployment scripts) --> B(Publish created scripts) --> E(Backup deployment state)
This is a single-stage workflow to destroy the deployed IaC resources of an environment and report any failures for easier cost and resource management. Note that the details of stage execution may vary based on features available on the orchestrator's IAC tool.
This workflow has a set of inputs variable that can be updated at the execution time. While the inputs could vary based on the selected IaC tool, one common input variable across all is the environment name, which defines what environment configurations to be used.
This is a feature that helps ensuring best practices in IAC code repos when development is in progress, Pull Request workflow offers a set of validations to vet the pull request branch code changes as part of the review process. This will ensure changes of the IaC have not only been validated at the code quality level but also been properly tested on a real deployed environment to see the impact of resource changes.
Symphony workflows use a set of tools to run tasks at each stage. Those tools are automatically installed and configured by the workflow tasks as needed on the SCM agent during the execution of the workflow tasks.
- Azure CLI is used to authenticate to the targeted environment Azure subscription. Here is a script to locally install it.
- Gitleaks is used to scan and prevent secrets from getting leaked into the main branch. Here is a script to locally install it. Some alternatives to Gitleaks are, Aqua Vulnerability Scanning and Management, GitHub Code Scanning, Mend SAST, Oxeye, Sonar Cloud
- Bicep is used to deploy the Bicep IaC modules. Here is a script to auto-install it.
- Powershell Core is used to execute PowerShell scripts, like ARM-TTK scripts. Here is a script to auto-install it.
- ARM-TTK is used to validate ARM templates. Here is a script to auto-install it.
- Shellspec is used to write and execute tests to test Bicep modules. Here is a script to auto-install it.
- Pester is used to write and execute tests to test Bicep modules. Here is a script to auto-install it.
- Terraform is used to deploy IaC modules resources. Here is a script to auto-install it.
- TFLint is used to validate the Terraform modules code. Here is a script to auto-install it.
- Go is used to write and execute tests for the terraform modules code using Terratest. Here is a script to auto-install it.