How our SRE Studio efficiently manages cloud infrastructures
Qubika’s SRE, Cloud, and Cybersecurity Studio share key tools they use to efficiently manage cloud infrastructures.
Based on tjtharrison’s solution.
One challenge we’ll face sooner or later when managing multiple Terraform modules is how to effectively organize them and how to handle Terraform module versioning. Should we opt for individual repositories or consolidate them into a monorepo?
Each approach has its advantages and drawbacks. Placing each module in its own repository ensures modularity and encapsulation, which simplifies the management of releases. However, this approach might lead to a complex web of dependencies when changes in one module impact several others.
In contrast, using a single repository could serve as a centralized source of truth for all infrastructure configurations, making the testing and review processes more efficient. The difficulty, however, will be in creating and managing versioned releases to allow deploying different versions of our modules on development and production environments. This is where careful Terraform module versioning becomes crucial.
In this post, our focus will be on utilizing automation and the semantic-release tool as a solution for managing releases and Terraform module versioning in a monorepo setup.
In this post, our focus will be on utilizing automation and the semantic-release tool as a solution for managing releases and Terraform module versioning in a monorepo setup.
Using the semantic-release tool
Semantic-release is a NodeJS tool that can be used on any code base to automate the whole package release workflow including determining the next version number, generating the release notes, and publishing the package.
It follows the standard Semantic Versioning format of MAJOR.MINOR.PATCH, for example 1.2.3. Using semantic-release in a project can help ensure that all versions created will strictly adhere to semantic versioning, prohibiting jumping versions or moving backwards.
How does semantic-release work?
Semantic-release uses commit messages to determine the impact of the pushed changes in the codebase. By following formalized conventions for commit messages, it automatically determines the next semantic version number, generates a changelog and publishes the release.
The table below shows which commit message gets you which release type when semantic-release runs (using the default configuration).
| Commit message | Release type |
|---|---|
| `fix(pencil): stop graphite breaking when too much pressure applied` | |
| `feat(pencil): add ‘graphiteWidth’ option` | |
| `perf(pencil): remove graphiteWidth option` `BREAKING CHANGE: The graphiteWidth option has been removed.` `The default graphite width of 10mm is always used for performance reasons.` |
Semantic-release is meant to be executed on the continuous integration environment after every successful build on the release branch. This way no human is directly involved in the release process.
Implementation
The Qubika’s sre-terraform-modules repository will be used as an example on how release automation could be implemented. The repository currently has the following structure:
├── .github
│ └── workflows
│ ├── main-release.yam
│ └── pr-terraformdocs.yaml
├── scripts
│ └── prep_modules.py
├── tf-aws-bastion
│ ...
│ ├── package.json
│ ...
...package.json
A package.json file must be added to each module folder. It will later be programmatically extended by a GitHub workflow when a new PR is merged to the main branch. This extended file will then be used by semantic-release to release a new version. Each file should only include the module name and a short description, as follows:
{
"name": "tf-aws-bastion",
"description": "AWS Bastion Terraform module"
}Main Release
The main-release.yaml workflow will run two jobs every time a PR is merged into the main branch. The first job uses the tj-actions/changed-files action to track all the changed files inside the modules folders and will generate a matrix with these
generate-matrix:
name: Generate matrix
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.get-matrix.outputs.all_changed_files }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Get matrix
id: get-matrix
uses: tj-actions/changed-files@v42
with:
path: ''
dir_names: true
json: true
escape_json: false
dir_names_exclude_current_dir: false
dir_names_max_depth: 1
files: ''
files_ignore: |
.github/**
scripts/**Want to learn more about our Cloud, SRE, DevOps & Cybersecurity Studio?
With our Cloud, SRE, and DevOps Studio embrace cloud-native solutions for accelerated development, combined with reliable, secure, and scalable environments.
The second job uses the matrix generated previously to release a new version of each modified folder. The most important steps in this job are the one that runs the Python script to prepare the module for deployment, and the final step that runs the semantic-release tool to generate a new release when needed based on the updated package.json files.
...
- name: Install dependencies
run: |
cd ${{ matrix.module }}
python3 "${GITHUB_WORKSPACE}"/scripts/prep_modules.py
npm install
- name: Release
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
cd ${{ matrix.module }}
npx semantic-release -t ${{ matrix.module }}/'${version}'terraform-docs
Another Github Actions workflow named pr-terraformdocs, will also be triggered with each merge into the main branch. This allows us to generate Terraform modules documentation using terraform-docs and gomplate.
The second job of the workflow utilizes the matrix created in the previous one, containing the names of the modified modules, to auto commit docs to an open pull request on the main branch.
...
- name: Render terraform docs inside the README.md and push changes back to PR branch
uses: terraform-docs/gh-actions@v1.0.0
with:
working-dir: ${{ env.modules_path }}/${{ matrix.module }}
output-file: README.md
output-method: inject
git-push: "true"
git-commit-message: "docs: update README.md with terraform-docs"Closing Notes
To create a new release of a module, commit your changes with the required message according to the type of changes, for instance for a patch use git commit -m fix: Updates variable type this would create a new release of type X.X.PATCH.
In conclusion, combining tools like GitHub Actions and semantic versioning, allows us to efficiently manage all our modules within a single repository and streamline Terraform module versioning. This approach simplifies the release process and ensures that each module can be independently versioned and deployed without hassle.
