Azure Devops Terraform Modules Monorepo

12 Nov 2023 | Tags: Azure DevOps, CICD, Terraform, Monorepo

• Introduction
• The Solution
  • Repository Prerequisites
  • The Monorepo Flow
    • Commit and Create a Pull Request
      • Linting
      • Security Analysis
      • Module Validation
    • Merge
  • Using the Module from monorepo
    • Using System.AccessToken to Download Module

Introduction

In my last blog post, I explored the concept of a GitHub-powered Terraform modules monorepo. However, I know many of you are using Azure DevOps for your CI/CD pipelines. So, I’ve come up with a pipeline solution that lets you manage and version your Terraform modules in either private or public Azure Repos.

The Solution

Getting this Azure DevOps-powered solution up and running wasn’t a walk in the park. I faced hurdles like missing runtime variables and difficulties passing data from pull requests to pipeline runs after merging. I’ve tackled similar challenges before and overcame these by making calls to the Azure DevOps API.

The solution is composed of the main .pipelines/ci.yaml, a pull request template, and three stage templates in the .pipelines/templates directory. This modular approach makes the pipeline easy to maintain, and you can customize it by adding the tools you prefer.

You can find the repository with the pipelines code here: Terraform Modules Monorepo on Azure DevOps

Repository Prerequisites

Before diving in, there are a few preparatory steps you need to take to ensure a smooth setup.

If your project security setting Limit job authorization scope to current project for non-release/release pipelines is configured to OFF then you will need to set up mentioned permissions for Project Collection Build Service (organization_name) instead of for project_name Build Service (organization_name). Job access token documentation

1. Import Code: You can either commit files from my repository or import them by pasting the URL https://github.com/krukowskid/terraform-modules-monorepo-on-azure-devops` in the import wizard.

   

2. Add Pipeline: Modify the name of the default branch in .pipelines/ci.yaml if needed.

   

    trigger:
      branches:
        include:
        - main # edit main
      ...
   
      variables:
        isMaster: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')] #edit refs/heads/main
      ...
   

3. Set build Validation for Default Branch: This triggers the monorepo pipeline for each PR targeted to the default branch.

   

4. Initialize Wiki and Set Wiki Permissions: Set up a wiki and configure permissions for project_name Build Service Account (organization).

   

5. Set Repository Permissions for Build Service: Assign necessary permissions for successful pipeline execution.

   

6. Turn Off Repository Access Protection: This is required for the Wiki, as the wiki resides in a different repository.

   

7. Limit Merge Types to Squash on Default Branch

   

The Monorepo Flow

Commit and Create a Pull Request

Fill in the changelog entry, which will appear in the PR description and autogenerated documentation. Assign a tag (major, minor, patch or no-release) to the PR to trigger the workflow and generate a new semantic version number.

When creating a pull request, the pull request template comes into play. At this stage, you should fill in a changelog entry, which will be visible in the PR description and autogenerated documentation. You should also assign a tagat this point to allow the workflow to generate a new semantic version number.

Several checks are performed. If you forget the changelog entry or tag, the workflow blocks the PR from merging.

Following the initial check and module detection, the workflow performs the following actions on each modified module:

Linting

Uses the tflint tool and configuration from .tflint.hcl. You may need to add additional plugins to the configuration. Default are terraform and azurerm

Security Analysis

Utilizes checkov and tfsec for security analysis.

Module Validation

Checks for formatting issues, runs terraform init and terraform validate. Workflow fails if any checks fail.

If all checks pass, the pipeline tags your PR with the changed modules.

It provides a release plan comment, detailing the new version numbers for each modified module and changelog information for post-merge autogenerated documentation.

Merge

After merging changes, the pipeline generates and publishes documentation to the repository wiki.

It also publishes a new version of the module by adding a tag to the repository.

Using the Module from monorepo

To use a module in your Terraform template, reference it like this:

module "module_reference_name" {
  source = "git::https://{org_name}@dev.azure.com/{org_name}/{project_name}/_git/{repo_name}?ref={module_name}/v{version}"
}

For example:

module "app_configuration" {
  source = "git::https://pkrukowski0304@dev.azure.com/pkrukowski0304/TerraformModules/_git/TerraformModules?ref=dns_zone/v1.0.0""
}

For testing changes from a branch during development:

module "app_configuration" {
  source = "git::https://pkrukowski0304@dev.azure.com/pkrukowski0304/TerraformModules/_git/TerraformModules//module_name?ref=branch_name"
}

If you intend to keep your modules in a private repository, ensure that your app, identity, or user has the proper permissions to access the repo. Terraform uses default git permissions when checking out modules. In local development, your git credentials will suffice, but in Azure Piplines, I use a system access token and set git to add an authorization header on requests to my monorepo repository

Using System.AccessToken to Download Module

If you create a repository in a separate project and use System.AccessToken, set up additional permissions for each project needing access.

Disable protection for the YAML pipeline and add project_name Build Service (organization_name) to the Readers group on your Terraform Monorepo project.

Then if you limit access from pipelines like on the screenshot below

you need to add project_name Build Service (organization_name) to Readers group on your Terraform Monorepo project

Example pipeline:

trigger:
- main

pool:
  vmImage: ubuntu-latest
  
variables:
  SYSTEM_ACCESSTOKEN: $(System.AccessToken)

steps:
- script: |
    git config --global http.https://dev.azure.com/pkrukowski0304/Terraform%20Modules.extraheader "Authorization: bearer $(System.AccessToken)"
  displayName: 'Setup TerraformModules access header'

  #terraform steps