GitHub-Powered Terraform Modules Monorepo
Introduction
In one of my recent projects, I found myself in a heated debate over whether to adopt a polyrepo (multiple repositories) or a monorepo (single repository) approach for housing all the Terraform modules utilized within our company. A single Git repository per module offers simplicity in version management, workflow creation, and lifecycle management. However, it can be challenging to keep track of numerous modules, and the task of creating documentation becomes more complex. Additionally, if your project already employs a monorepo for application code and templates, it can seem counterintuitive to create dozens of repositories, especially for a few lines of code in a single Terraform module.
In my quest for a solution, I scoured the web for ready-made answers but didn’t find an exact match for my requirements. Fortunately, I stumbled upon a project with similar constraints to mine: https://github.com/freight-hub/terraform-modules-demo which became a starting point for my solution.
The ingenious minds at Freight Hub had crafted a workflow that handled versioning modules from a monorepo and uploading them to AWS S3. It was my starting point for this project. However, I had my reasons for wanting to keep my modules in a private GitHub repository, and I also wished to incorporate validation tools like linting, tfsec, and checkov to ensure that changes were thoroughly tested before merging into the main branch and bumping the version number.
The Solution
It took me some time to create those workflows, and I even asked for assistance of a colleague with whom I had debated the best approach for handling modules. The solution comprises two primary GitHub workflows, a pull request template, and five “helper” workflows designed to improve readability and maintability compared to consolidating everything into a single workflow file.
You can find example repository under this link: https://github.com/krukowskid/terraform-modules-monorepo-on-github
Repository Prerequisites
To make this solution work, you’ll need to take a few preparatory steps.
Create an Empty Wiki Page
To enable the workflow to commit to the wiki and maintain autogenerated documentation, you must create at least one wiki page. Otherwise, the wiki workflow will fail because it won’t locate the wiki.
Grant Read and Write Access for Workflows in the Repository
You’ll find this section under the settings tab in your repository.
Add Labels
You’ll need four labels for pull requests:
majorminorpatchno-release
Add GitHub Application and Set Up Secrets (Optional)
If your Terraform modules reference other reusable modules within the same repository, you’ll need to set up a GitHub application with repository permissions and create repository secrets named appId and appPrivateKey. The workflow should automatically detect these secrets and authenticate with the git repository without additional steps.
Branch Protection Rules (Optional)
While these settings are optional, I recommend them as a starting point.
The Monorepo Flow
Commit and Create a Pull Request
When creating a pull request, the pull request template comes into play. At this stage, you should fill in a changelog entry, which will be visible in the PR description and autogenerated documentation. You should also assign a label at this point to allow the workflow to generate a new semantic version number.
After creating the pull request, several checks are performed. If you forget to label your PR or you add more than one label, the workflow will fail after the initial check.
Following the initial check and detection of changed modules, the workflow carries out the following actions on each modified module:
Linting
This step employs the tflint tool and configuration from .tflint.hcl. You may need to add additional plugins to the configuration. In my example, I’m using terraform and azurerm plugins.
Security Analysis
Two tools, checkov and tfsec, are used for security analysis. These tools have proven to work well together.
Module Validation
The changed module is checked for formatting issues; unformatted files will result in a workflow failure. Afterward, it runs terraform init and terraform validate.
If any of the above checks fail, it triggers a failed workflow, signaling that something is amiss with your proposed changes.
If all checks pass, the Action will label your PR with the changed modules and provide a release plan comment, which describes new version number and changelog information.
Merge
After merging changes from your PR, the action will publish documentation to the repository wiki.
It will also publish a new version of the module by adding a tag to the repository. Before tagging, it extracts only the tagged module files so that within a tag, you’ll see only those files.
Additionally, the major and major.minor versions are updated. I always recommend using exact versions, but I understand that in some cases, this approach might work.
Using the Module
To use a module in your Terraform template, you need to reference it in the following way:
module "module_reference_name" {
source = "git::https://github.com/{username_or_org_name}/{repo_name}.git?ref={module_name}/v{version}"
}
For example, for my repository, it would be:
module "app_configuration" {
source = "git::https://github.com/krukowskid/terraform-modules-monorepo-on-github.git?ref=app_configuration/v1.0.1"
}
If you want to test your changes from a branch during development, you can do it this way:
module "app_configuration" {
source = "git::https://github.com/krukowskid/terraform-modules-monorepo-on-github.git//module_name?ref=branch_name"
}
If you intend to keep your modules in a private repository, ensure that your app, identity, or user has the proper permissions to access the repo. Terraform uses default git permissions when checking out modules. In local development, your git credentials will suffice, but in GitHub Actions, I use a GitHub Application and two additional steps to authenticate with the git repo:
- name: Generate token
env:
APP_ID: ${{ secrets.appId }}
PRIVATE_KEY: ${{ secrets.appPrivateKey }}
if: ${{ env.APP_ID != null && env.PRIVATE_KEY != null }}
id: generate_token
uses: tibdex/github-app-token@v1
with:
app_id: ${{ secrets.appId }}
private_key: ${{ secrets.appPrivateKey }}
- name: Authenticate to git with bot token
env:
APP_ID: ${{ secrets.appId }}
PRIVATE_KEY: ${{ secrets.appPrivateKey }}
if: ${{ env.APP_ID != null && env.PRIVATE_KEY != null }}
run: |
echo ${{ steps.generate_token.outputs.token }} | gh auth login --with-token
gh auth setup-git
Conclusion
While the debate between monorepos and polyrepos continues, the advantages of a monorepo in terms of centralized management, documentation, collaboration, and simplified workflows make it a compelling choice for organizations looking to optimize their Terraform module development process. Ultimately, the decision should align with the specific needs and goals of the project, but the monorepo approach offers undeniable benefits.
