Header

Introduction

In one of my recent projects, I found myself in a heated debate over whether to adopt a polyrepo (multiple repositories) or a monorepo (single repository) approach for housing all the Terraform modules utilized within our company. A single Git repository per module offers simplicity in version management, workflow creation, and lifecycle management. However, it can be challenging to keep track of numerous modules, and the task of creating documentation becomes more complex. Additionally, if your project already employs a monorepo for application code and templates, it can seem counterintuitive to create dozens of repositories, especially for a few lines of code in a single Terraform module.

In my quest for a solution, I scoured the web for ready-made answers but didn’t find an exact match for my requirements. Fortunately, I stumbled upon a project with similar constraints to mine: https://github.com/freight-hub/terraform-modules-demo which became a starting point for my solution.

The ingenious minds at Freight Hub had crafted a workflow that handled versioning modules from a monorepo and uploading them to AWS S3. It proved to be immensely helpful. However, I had my reasons for wanting to keep my modules in a private GitHub repository, and I also wished to incorporate validation tools like linting, tfsec, and checkov to ensure that changes were thoroughly tested before merging into the main branch and bumping the version number.

The Solution

It took me some time to devise this workflow, and I even sought the assistance of a colleague with whom I had debated the best approach for handling modules. The solution comprises two primary GitHub workflows, a pull request template, and five “helper” workflows designed to enhance readability compared to consolidating everything into a single workflow file.

You can find example repository under this link: https://github.com/krukowskid/terraform-modules-monorepo-on-github

Repository Prerequisites

To make this solution work, you’ll need to take a few preparatory steps.

Create an Empty Wiki Page

Create the first wiki page

To enable the workflow to commit to the wiki and maintain autogenerated documentation, you must create at least one wiki page. Otherwise, the wiki workflow will fail because it won’t locate the wiki.

Grant Read and Write Access for Workflows in the Repository

Workflow permissions

You’ll find this section under the settings tab in your repository.

Add Labels

You’ll need four labels for pull requests:

  • major
  • minor
  • patch
  • no-release

Add GitHub Application and Set Up Secrets (Optional)

If your Terraform modules reference other reusable modules within the same repository, you’ll need to set up a GitHub application with repository permissions and create repository secrets named appId and appPrivateKey. The workflow should automatically detect these secrets and authenticate with the git repository without additional steps.

Branch Protection Rules (Optional)

Branch protection rules view

While these settings are optional, I recommend them as a starting point.

The Monorepo Flow

Commit and Create a Pull Request

When creating a pull request, the pull request template comes into play. At this stage, you should fill in a changelog entry, which will be visible in the PR description and autogenerated documentation. You should also assign a label at this point to allow the workflow to generate a new semantic version number.

Create PR with label

After creating the pull request, several checks are performed. If you forget to label your PR or you add more than one label, the workflow will fail after the initial check.

Failure because of label missing

Following the initial check and detection of changed modules, the workflow carries out the following actions on each modified module:

Linting

This step employs the tflint tool and configuration from .tflint.hcl. You may need to add additional plugins to the configuration. In my example, I’m using terraform and azurerm plugins.

Security Analysis

Two tools, checkov and tfsec, are used for security analysis. These tools have proven to work well together.

Module Validation

The changed module is checked for formatting issues; unformatted files will result in a workflow failure. Afterward, it runs terraform init and terraform validate.

If any of the above checks fail, it triggers a failed workflow, signaling that something is amiss with your proposed changes.

checkov-fail

If all checks pass, the Action will label your PR with the changed modules and provide a release plan comment, which describes new version number and changelog information.

PR labeled by GitHub Action Release plan comment

Merge

After merging changes from your PR, the action will publish documentation to the repository wiki.

checkov-fail

It will also publish a new version of the module by adding a tag to the repository. Before tagging, it extracts only the tagged module files so that within a tag, you’ll see only those files.

Tagged module files

Additionally, the major and major.minor versions are updated. I always recommend using exact versions, but I understand that in some cases, this approach might work.

Tagged module files

Using the Module

To use a module in your Terraform template, you need to reference it in the following way:

module "module_reference_name" {
  source = "git::https://github.com/{username_or_org_name}/{repo_name}.git?ref={module_name}/v{version}"
}

For example, for my repository, it would be:

module "app_configuration" {
  source = "git::https://github.com/krukowskid/terraform-modules-monorepo-on-github.git?ref=app_configuration/v1.0.1"
}

If you want to test your changes from a branch during development, you can do it this way:

module "app_configuration" {
  source = "git::https://github.com/krukowskid/terraform-modules-monorepo-on-github.git//module_name?ref=branch_name"
}

If you intend to keep your modules in a private repository, ensure that your app, identity, or user has the proper permissions to access the repo. Terraform uses default git permissions when checking out modules. In local development, your git credentials will suffice, but in GitHub Actions, I use a GitHub Application and two additional steps to authenticate with the git repo:

- name: Generate token
  env:
    APP_ID: ${{ secrets.appId }}
    PRIVATE_KEY: ${{ secrets.appPrivateKey }}
  if: ${{ env.APP_ID != null && env.PRIVATE_KEY != null }}
  id: generate_token
  uses: tibdex/github-app-token@v1
  with:
    app_id: ${{ secrets.appId }}
    private_key: ${{ secrets.appPrivateKey }}

- name: Authenticate to git with bot token
  env:
    APP_ID: ${{ secrets.appId }}
    PRIVATE_KEY: ${{ secrets.appPrivateKey }}
  if: ${{ env.APP_ID != null && env.PRIVATE_KEY != null }}
  run: |
    echo ${{ steps.generate_token.outputs.token }} | gh auth login --with-token
    gh auth setup-git

Conclusion

While the debate between monorepos and polyrepos continues, the advantages of a monorepo in terms of centralized management, documentation, collaboration, and streamlined workflows make it a compelling choice for organizations looking to optimize their Terraform module development process. Ultimately, the decision should align with the specific needs and goals of the project, but the monorepo approach offers undeniable benefits.