Cloudformation Pattern

Date: July 20, 2019 Author: Brian Hooper

TLDR

This post is a quick walk-through of a solution pattern I often use for new AWS Cloudformation Projects.


Introduction

AWS Cloudformation is an infrastructure as code solution that provides a consistent workflow to model a collection of related AWS resources, provision and manage them throughout their lifecycles via declarative templates and configuration files.


Structure

Here is a base structure for using a stacks-based approach with your Cloudformation projects:

    /project                        # Cloudformation Project
    |
    |__ /bin                        # Project Binaries 
    |
    |__ /scripts                    # Project Scripts for CI/CD Processes
    |
    |__ /stacks                     # Cloudformation Stacks
    |   |      
    |   |__ /stack1                 # Stack 1 (basic example)
    |   |   |
    |   |   |__ /params             # Cloudformation Stack Parameters
    |   |   |   |__ default.json    # Default Params to be used when environment specific params does not exist
    |   |   |   |__ dev.json        # Environment specific param files (dev|tst|stg|prd).json and etc
    |   |   |   |__ tst.json
    |   |   |   |__ stg.json
    |   |   |   |__ prd.json
    |   |   | 
    |   |   |__ /policy             # Cloudformation Stack Policy
    |   |   |   |__ default.json    # Default Policy to be used when environment specific policy does not exist
    |   |   |   |__ dev.json        # Environment specific policy files (dev|tst|stg|prd).json and etc
    |   |   |   |__ tst.json
    |   |   |   |__ stg.json
    |   |   |   |__ prd.json
    |   |   |
    |   |   |__ README.md           # Cloudformation Stack Documentation 
    |   |   |__ template.yaml       # Cloudformation Stack Template
    |   |  
    |   |__ /stack2                 # Stack 2
    |   |__ /stack3                 # Stack 3
    |   |__ /stackN                 # Stack N
    |
    |__ cloudformation.sh           # Cloudformation Wrapper

Basic Stack Structure

Here is a “basic” stack example:

    |__ /stack                  # Stack (basic example)
    |   |
    |   |__ /params             # Cloudformation Stack Parameters
    |   |   |__ default.json    # Default Params to be used when environment specific params does not exist
    |   |   |__ dev.json        # Environment specific param files (dev|tst|stg|prd).json and etc
    |   |   |__ tst.json
    |   |   |__ stg.json
    |   |   |__ prd.json
    |   | 
    |   |__ /policy             # Cloudformation Stack Policy
    |   |   |__ default.json    # Default Policy to be used when environment specific policy does not exist
    |   |   |__ dev.json        # Environment specific policy files (dev|tst|stg|prd).json and etc
    |   |   |__ tst.json
    |   |   |__ stg.json
    |   |   |__ prd.json
    |   |
    |   |__ README.md           # Cloudformation Stack Documentation 
    |   |__ template.yaml       # Cloudformation Stack Template

Application Stack (SAM)

Here is an “application” stack example:

    |__ /stack                  # Stack (app example)
    |   |
    |   |__ /params             # Cloudformation Stack Parameters
    |   |   |__ default.json    # Default Params to be used when environment specific params does not exist
    |   |   |__ dev.json        # Environment specific param files (dev|tst|stg|prd).json and etc
    |   |   |__ tst.json
    |   |   |__ stg.json
    |   |   |__ prd.json
    |   | 
    |   |__ /policy             # Cloudformation Stack Policy
    |   |   |__ default.json    # Default Policy to be used when environment specific policy does not exist
    |   |   |__ dev.json        # Environment specific policy files (dev|tst|stg|prd).json and etc
    |   |   |__ tst.json
    |   |   |__ stg.json
    |   |   |__ prd.json
    |   |
    |   |__ /scripts            # Application Scripts (build|lint|test|deploy)
    |   |
    |   |__ /source             # Application Source Code
    |   |
    |   |__ README.md           # Cloudformation Stack Documentation 
    |   |__ template.yaml       # Cloudformation Stack Template (SAM)

Components

Here are the core components of this stack pattern:

  • Stack Parameters Files
  • Stack Policy Files
  • Stack README
  • Stack Template

Stack Parameters File

Here is an example stack parameter file (e.g. default.json )

[
    {
        "ParameterKey": "pName",
        "ParameterValue": "MyName"
    },
    {
        "ParameterKey": "pStack",
        "ParameterValue": "MyStack"
    },
    {
        "ParameterKey": "pWorkspace",
        "ParameterValue": "dev"
    }
]

Stack Policy File

Here is an example stack policy file (e.g. default.json )

{
    "Statement" : [
        {
        "Effect" : "Allow",
        "Action" : ["Update:*"],
        "Principal": "*",
        "Resource" : "*"
        }
    ]
}

README.md

Here is an example stack README.md file

# AWS CloudFormation <Name_Of_Template> Template

## Table of contents

* [Summary](#summary)
* [Usage](#usage)
* [Components](#components)
* [Structure](#structure)
* [Issues](#issues)
* [Contributions](#contributions)
* [Questions](#questions)

## Summary

Author(s): Author Name (@Alias)
Email(s): Email Address

Use this section to provide a detailed overview of your template.

* What use case(s) or problem(s) does your solution pattern solve?
* What resources does your template provision?

## Usage

Examples of how to use this template:

## Components

Use this section to provide a detailed review of the core components of your template.

* Describe the 'resources' that are deployed using this template
* Describe the dependencies between resources
* Describe the default configuration of the resources
* Describe any special policy considerations (UpdatePolicy, DeletePolicy, etc)

## Structure

CloudFormation templates describe your AWS resources and their properties.  

Use this section to describe the structure of your Service Template:

    /stack                  # Stack Template Root.
            
        /parameters         # Parameters Directory for the CloudFormation Template
            default.json    # Default Parameters to be used when Environment Specific Parameters File Does Not Exist
                            # Environment Specific Parameter Files can be implemented via dev.json, tst.json, prd.json

        /policy             # Policy Directory for the CloudFormation Stack Policy
            default.json    # Default Stack Policy to be used when Environment Specific Policy File Does Not Exist
                            # Environment Specific Policy Files can be implemented via dev.json, tst.json, prd.json
            
        service.yaml        # Service Template Configuration File
        template.yaml       # CloudFormation Template for the Service
        README.md           # README File describing the service template

## Issues

Describe how users should report any issues that are identified while using this Service Template

Example:

If you identify an issue using this template please contact support@example.com

Detailed information is very helpful to understand and prioritize issues.

    How to reproduce the issue, step-by-step.
    The expected behavior (or what is wrong).
    The error message(s)
    The operating system
    Screenshots if applicable

## Contributions

Pull Requests (PRs) are always welcome.  Please create a new branch and submit a pull-request for review.

## Questions

Still have questions?  Please contact support@example.com

Stack Template

Here is an example stack template.yaml file:

---
# CloudFormation Stack Template

AWSTemplateFormatVersion: "2010-09-09"
 
Description: >
  Here is a description and 
  some additional details
  about this template.   
Metadata:
  template metadata

Parameters:
  pName:
    Description: Sets the Name. Default is <blank>
    Type: String
    Default: <blank>
  pStack:
    Description: Sets the Stack. Default is <blank>
    Type: String
    Default: <blank>
  pWorkspace:
    Description: Sets the Workspace. Default is dev.
    Type: String
    Default: dev
    ConstraintDescription: Must be an allowed value. (dev|tst|stg|prd|ops)
    AllowedValues: 
      - dev     # Development
      - tst     # Testing
      - stg     # Staging
      - prd     # Production
      - ops     # Operations 

Mappings:
  set of mappings

Conditions:
  set of conditions

Transform:
  set of transforms

Resources:
  set of resources

Outputs:
  set of outputs

Workflow

Here is a sample Cloudformation Wrapper script for a more programmatic workflow, that uses JQ to parse the JSON parameter and policy files. The Cloudformation wrapper script performs the following basic actions:

  • List Stack
  • List Parameters
  • Describe Stack
  • Describe Stacks
  • Deploy Stack
  • Destroy Stack
  • Stage Stack
  • Validate Template
#!/usr/bin/env bash

## ARGUMENTS
## ====================================
ACTION=$1
STACK=$2
WORKSPACE=$3
if [ -z "$WORKSPACE" ]
then
      echo "Workspace argument not specified, using 'default'"
      WORKSPACE="default"
fi

## FUNCTIONS
## ====================================
function help () {
    echo "Description:"
    echo "    CloudFormation Script is a simple programmatic workflow"
    echo ""
    echo "Usage: "
    echo "    ./cloudformation.sh <ACTION> <STACK> <WORKSPACE>"
    echo ""
    echo "Example:"
    echo "    ./cloudformation.sh list-stacks"
    echo "    ./cloudformation.sh list-stack s3-bucket"
    echo "    ./cloudformation.sh list-parameters s3-bucket dev"
    echo "    ./cloudformation.sh deploy-stack s3-bucket dev"
    echo "    ./cloudformation.sh describe-stack s3-bucket"
    echo "    ./cloudformation.sh describe-stacks"
    echo "    ./cloudformation.sh destroy-stack s3-bucket dev"
    echo "    ./cloudformation.sh stage-stack s3-bucket dev"
    echo "    ./cloudformation.sh validate-template s3-bucket"
    echo ""
}

function describeStack () {
    echo ""
    echo "CloudFormation Script is Describing Stack '$STACK' in $WORKSPACE"
    COMMAND="aws cloudformation describe-stacks --stack-name $STACK "
    echo $COMMAND
    eval $COMMAND
}

function describeStacks () {
    echo ""
    echo "CloudFormation Script is Describing Stacks"
    COMMAND="aws cloudformation describe-stacks"
    echo $COMMAND
    eval $COMMAND
}

function stageStack () {
    echo ""
    echo "CloudFormation Script is getting Parameters for $STACK in $WORKSPACE "
    PARAMS=$(jq -r '.[] | [.ParameterKey, .ParameterValue] | "\(.[0])=\"\(.[1])\""' ./stacks/$STACK/params/$WORKSPACE.json)
    echo "-------------------------------------"
    echo "./stacks/$STACK/params/$WORKSPACE.json"
    echo "-------------------------------------"
    echo "$PARAMS"

    echo ""
    echo "CloudFormation Script is Staging Stack '$STACK' in $WORKSPACE "
    COMMAND="aws cloudformation deploy --stack-name $STACK --template-file ./stacks/$STACK/template.yaml --no-execute-changeset --parameter-overrides $PARAMS"
    echo $COMMAND
    echo ""
    eval $COMMAND
}

function deployStack () {
    echo ""
    echo "CloudFormation Script is getting Parameters for $STACK in $WORKSPACE "
    PARAMS=$(jq -r '.[] | [.ParameterKey, .ParameterValue] | "\(.[0])=\"\(.[1])\""' ./stacks/$STACK/params/$WORKSPACE.json)
    echo "-------------------------------------"
    echo "./stacks/$STACK/params/$WORKSPACE.json"
    echo "-------------------------------------"
    echo "$PARAMS"

    echo ""
    echo "CloudFormation Script is Deploying Stack '$STACK' in $WORKSPACE "
    COMMAND="aws cloudformation deploy --stack-name $STACK --template-file ./stacks/$STACK/template.yaml --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --parameter-overrides $PARAMS"
    echo $COMMAND
    echo ""
    eval $COMMAND
}

function destroyStack () {
    echo ""
    echo "CloudFormation Script is Destroying Stack '$STACK' in $WORKSPACE"
    COMMAND="aws cloudformation delete-stack --stack-name $STACK "
    echo $COMMAND
    eval $COMMAND
}

function describeStacks () {
    echo ""
    echo "CloudFormation Script is Describing Stacks"
    COMMAND="aws cloudformation describe-stacks"
    echo $COMMAND
    eval $COMMAND
}

function describeStack () {
    echo ""
    echo "CloudFormation Script is Describing Stack '$STACK'"
    COMMAND="aws cloudformation describe-stack --stack-name $STACK"
    echo $COMMAND
    eval $COMMAND
}

function listParameters () {
    echo ""
    echo "CloudFormation Script is Listing Parameters for Stack '$STACK' in $WORKSPACE"
    PARAMS=$(jq -r '.[] | [.ParameterKey, .ParameterValue] | "\(.[0])=\"\(.[1])\""' ./stacks/$STACK/params/$WORKSPACE.json)
    echo "-------------------------------------"
    echo "./stacks/$STACK/params/$WORKSPACE.json"
    echo "-------------------------------------"
    echo $PARAMS
    echo ""
}

function validateTemplate () {
    echo ""
    echo "CloudFormation Script is Validating Template for Stack '$STACK'"
    COMMAND="aws cloudformation validate-template --template-body file://./stacks/$STACK/template.yaml"
    echo $COMMAND
    eval $COMMAND
}

case "$ACTION" in
        stage-stack)
            stageStack
            ;;
        deploy-stack)
            deployStack
            ;;
        describe-stack)
            describeStack
            ;;
        describe-stacks)
            describeStacks
            ;;    
        destroy-stack)
            destroyStack
            ;;
        list-stack)
            describeStack
            ;;
        list-stacks)
            describeStacks
            ;;
        list-parameters)
            listParameters
            ;;
        validate-template)
            validateTemplate
            ;;        
        *)
            help
            ;;
esac

Next

The above structure and workflow are useful for initializing new cloudformation projects and maintaining them over time as they scale. The workflow via the cloudformation.sh wrapper is beneficial because it standardizes and reinforces a consistent structure. As an architect and consultant I am often collaborating with different customers and teams. This approach is especially helpful when working across multiple projects or rapidly prototyping new solutions

After using this basic pattern in practice for several AWS Cloud projects I have designed and developed an approach for building a “Pattern Library” as a mechanism for scaling knowledge across projects and teams. I am working on an open source project called Supply that can be used to implement similar patterns and approaches. More to come!


Resources


© Traveler Theme 2023
Let's Stay Connected

Handcrafted by a @KnownTraveler