Terraform Pattern

TLDR

This post is a quick walk-through of a solution pattern I often use for new Terraform Projects.

Introduction

Terraform is an open-source infrastructure as code software tool that provides a consistent workflow to manage cloud services by codifying cloud APIs into declarative configuration files.

For effective collaboration, it’s important to delegate and empower teams to work in parallel without conflict. It is best to use a modular approach as it provides flexibility over time for maintaining a solution while improving the ability to manage the blast radius of changes as well as drift across workspaces and module versions. In practice I have found that it is best to follow: One Workspace per Environment per Terraform Configuration while using modules to ensure consistency.

A module is a container for multiple resources that are used together. They provide a single code base where all resources regardless of environment are provisioned with the same code ensuring that changes are promoted in a predictable way. Use modules as a level of abstraction, but be careful to avoid unecessary complexity. Lastly, DO NOT write modules that are simply thin wrappers around single resource types, just use the resource type directly in those cases.

Structure

Here is a base structure for using a modular approach with your Terraform projects:

    /project                    # Terraform Project
    |
    |__ /bin                    # Project Binaries 
    |   |
    |   |__ terraform           # Locally Source Terraform (Used by Wrapper)
    |
    |__ /modules                # Locally Sourced Modules (Organized by Provider)
    |   |
    |   |__ /acme
    |   |__ /aws
    |   |__ /gitlab
    |   |__ /grafana
    |   |__ /rundeck
    |   |__ /custom             # Custom Modules
    |
    |__ /scripts                # Custom Scripts for CI/CD Processes
    |   |
    |   |__ build.sh
    |   |__ deploy.sh
    |   |__ destroy.sh
    |   |__ drift.sh
    |   |__ lint.sh
    |   |__ test.sh
    |
    |__ /source                 # Application Source Code
    |            
    |__ /workspaces             # Workspaces (Organized by Account/Environment)
    |   |
    |   |__ /dev                # Development Workspace
    |   |   |__ /application
    |   |   |__ /database
    |   |   |__ /network
    |   |   |__ /security
    |   |   |__ /storage
    |   |__ /tst                # Testing Workspace
    |   |   |...
    |   |__ /stg                # Staging Workspace
    |   |   |...
    |   |__ /prd                # Production Workspace
    |   |   |...
    |
    |__ terraform.sh            # Terraform Wrapper

Workflow

The terraform.sh wrapper is a helpful mechanism for your local workspace to simplify and standardize the workflow:

• Manages the local Terraform Version (/bin)
• Installs any custom binaries (/bin)
• Manages Terraform Stack Initialization
• Manages Terraform Workspace Selection
• Runs Native Terraform Commands using the following arguments:
  • Workspace
  • Stack
  • Action

Usage

./terraform.sh <workspace> <stack> <action> <args>

# Examples
./terraform.sh dev my-stack init
./terraform.sh dev my-stack plan
./terraform.sh dev my-stack apply

Script

terraform.sh

#!/usr/bin/env bash

## CONSTANTS
## ====================================
TERRAFORM_VERSION=0.12.0
PROJECT_PATH=$(dirname $(readlink -f $0))
GLOBAL_PATH=$PROJECT_PATH/global
OS_SHORTNAME=$(uname | awk '{print tolower($0)}')


## ARGUMENTS
## ====================================
WORKSPACE=$1
STACK=$2
ACTION=$3
ACTION_ARGS=$(echo "$@" | awk '{first = $1; second = $2; third = $3; $1 = ""; $2 = ""; $3 = ""; print $0 }')

set -e

## TERRAFORM VERSION (LOCAL)
## ====================================
CURRENT_VERSION=$($PROJECT_PATH/bin/terraform --version 2>/dev/null | head -1 | awk '{print $2}' | sed 's/^v//')
if [ ! "$CURRENT_VERSION" == "$TERRAFORM_VERSION" ]; then
FILE=$(mktemp)
echo "This project is currently using terraform v$TERRAFORM_VERSION. You're at v$CURRENT_VERSION. Installing terraform@${TERRAFORM_VERSION}..."
if [ "$(uname)" == "Darwin" ]; then
    curl -L https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_darwin_amd64.zip -o $FILE
    unzip -o $FILE -d /tmp terraform
    mv /tmp/terraform $PROJECT_PATH/bin/terraform
    rm -rf $FILE
elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then
    curl -L https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip -o $FILE
    unzip -o $FILE -d /tmp terraform
    mv /tmp/terraform $PROJECT_PATH/bin/terraform
    rm -rf $FILE
elif [ "$(expr substr $(uname -s) 1 10)" == "MINGW64_NT" ]; then
    echo "Appears that you are running windows. Have you tried turning it off an on again?"
fi
fi


## TERRAFORM WRAPPER
## ====================================

echo ""
echo "=============================================================================="
echo " Executing terraform $ACTION against the $STACK stack in $WORKSPACE. "
echo "=============================================================================="
echo ""

## EXPORT VARIABLES
export STACK
export TARGET_DIR=$PROJECT_PATH/workspaces/$WORKSPACE/$STACK

## VERIFY TARGET DIRECTORY EXISTS (e.g. STACK)
if [ ! -d "$TARGET_DIR" ]; then
echo "Error: Stack \"$STACK\" does not exist in workspace \"$WORKSPACE\"."
exit 1
fi

## CHANGE TO TARGET DIRECTORY
cd $TARGET_DIR

## FORMAT TERRAFORM COMMAND
export CMD="$PROJECT_PATH/bin/terraform $ACTION $ACTION_ARGS"

## VERIFY TERRAFORM STACK IS INITIALIZED
if [ ! -d "$PROJECT_PATH/workspaces/$WORKSPACE/$STACK/.terraform" ]; then
echo "initalizing stack"
$PROJECT_PATH/bin/terraform init
fi

## GET UPDATES FOR TERRAFORM MODULES 
$PROJECT_PATH/bin/terraform get -update

set +e

## SELECT TERRAFORM WORKSPACE, ELSE CREATE WORKSPACE
$PROJECT_PATH/bin/terraform workspace select $WORKSPACE >/dev/null 2>&1
if [ "$?" -eq "1" ]; then
echo "creating new workspace: $WORKSPACE for stack $STACK"
$PROJECT_PATH/bin/terraform workspace new $WORKSPACE
fi

set -e

## TERRAFORM COMMAND
$CMD

Next

The above structure and workflow are useful for initializing new terraform projects and maintaining them over time as they scale. The workflow via the terraform.sh wrapper is beneficial because it standardizes and reinforces a consistent structure as well as manages the version of terraform in use. As an architect and consultant I am often collaborating with different customers and teams. This approach is especially helpful when working across multiple projects with differing versions of Terraform or rapidly prototyping new solutions.

After using this basic pattern in practice for several AWS Cloud projects I have designed and developed an approach for building a “Pattern Library” as a mechanism for scaling knowledge across projects and teams. I am working on an open source project called Supply that can be used to implement similar patterns and approaches. More to come!

Resources

• Terraform Website
• Terraform Best Practices
• Atlantis
• Terraboard
• Terragrunt

Updates

12/20/2020

Many teams use open source approaches such as custom wrapper scripts, custom CI/CD Pipelines, Atlantis, Terraboard, Terragrunt and etc to implement governance mechanisms for their Infrastructure as Code and Cloud Native projects. Since the original date of this post Hashicorp has continued to improve and evolve a number of services to support Terraform such as the Terraform Registry and Terraform Cloud.

At the time of this update you can sign up for Terraform Cloud for FREE ($0 up to 5 Users).