Cloudformation Pattern
Date: July 20, 2019 Author: Brian HooperTLDR
This post is a quick walk-through of a solution pattern I often use for new AWS Cloudformation Projects.
Introduction
AWS Cloudformation is an infrastructure as code solution that provides a consistent workflow to model a collection of related AWS resources, provision and manage them throughout their lifecycles via declarative templates and configuration files.
Structure
Here is a base structure for using a stacks-based approach with your Cloudformation projects:
/project # Cloudformation Project
|
|__ /bin # Project Binaries
|
|__ /scripts # Project Scripts for CI/CD Processes
|
|__ /stacks # Cloudformation Stacks
| |
| |__ /stack1 # Stack 1 (basic example)
| | |
| | |__ /params # Cloudformation Stack Parameters
| | | |__ default.json # Default Params to be used when environment specific params does not exist
| | | |__ dev.json # Environment specific param files (dev|tst|stg|prd).json and etc
| | | |__ tst.json
| | | |__ stg.json
| | | |__ prd.json
| | |
| | |__ /policy # Cloudformation Stack Policy
| | | |__ default.json # Default Policy to be used when environment specific policy does not exist
| | | |__ dev.json # Environment specific policy files (dev|tst|stg|prd).json and etc
| | | |__ tst.json
| | | |__ stg.json
| | | |__ prd.json
| | |
| | |__ README.md # Cloudformation Stack Documentation
| | |__ template.yaml # Cloudformation Stack Template
| |
| |__ /stack2 # Stack 2
| |__ /stack3 # Stack 3
| |__ /stackN # Stack N
|
|__ cloudformation.sh # Cloudformation Wrapper
Basic Stack Structure
Here is a “basic” stack example:
|__ /stack # Stack (basic example)
| |
| |__ /params # Cloudformation Stack Parameters
| | |__ default.json # Default Params to be used when environment specific params does not exist
| | |__ dev.json # Environment specific param files (dev|tst|stg|prd).json and etc
| | |__ tst.json
| | |__ stg.json
| | |__ prd.json
| |
| |__ /policy # Cloudformation Stack Policy
| | |__ default.json # Default Policy to be used when environment specific policy does not exist
| | |__ dev.json # Environment specific policy files (dev|tst|stg|prd).json and etc
| | |__ tst.json
| | |__ stg.json
| | |__ prd.json
| |
| |__ README.md # Cloudformation Stack Documentation
| |__ template.yaml # Cloudformation Stack Template
Application Stack (SAM)
Here is an “application” stack example:
|__ /stack # Stack (app example)
| |
| |__ /params # Cloudformation Stack Parameters
| | |__ default.json # Default Params to be used when environment specific params does not exist
| | |__ dev.json # Environment specific param files (dev|tst|stg|prd).json and etc
| | |__ tst.json
| | |__ stg.json
| | |__ prd.json
| |
| |__ /policy # Cloudformation Stack Policy
| | |__ default.json # Default Policy to be used when environment specific policy does not exist
| | |__ dev.json # Environment specific policy files (dev|tst|stg|prd).json and etc
| | |__ tst.json
| | |__ stg.json
| | |__ prd.json
| |
| |__ /scripts # Application Scripts (build|lint|test|deploy)
| |
| |__ /source # Application Source Code
| |
| |__ README.md # Cloudformation Stack Documentation
| |__ template.yaml # Cloudformation Stack Template (SAM)
Components
Here are the core components of this stack pattern:
- Stack Parameters Files
- Stack Policy Files
- Stack README
- Stack Template
Stack Parameters File
Here is an example stack parameter file (e.g. default.json )
[
{
"ParameterKey": "pName",
"ParameterValue": "MyName"
},
{
"ParameterKey": "pStack",
"ParameterValue": "MyStack"
},
{
"ParameterKey": "pWorkspace",
"ParameterValue": "dev"
}
]
Stack Policy File
Here is an example stack policy file (e.g. default.json )
{
"Statement" : [
{
"Effect" : "Allow",
"Action" : ["Update:*"],
"Principal": "*",
"Resource" : "*"
}
]
}
README.md
Here is an example stack README.md file
# AWS CloudFormation <Name_Of_Template> Template
## Table of contents
* [Summary](#summary)
* [Usage](#usage)
* [Components](#components)
* [Structure](#structure)
* [Issues](#issues)
* [Contributions](#contributions)
* [Questions](#questions)
## Summary
Author(s): Author Name (@Alias)
Email(s): Email Address
Use this section to provide a detailed overview of your template.
* What use case(s) or problem(s) does your solution pattern solve?
* What resources does your template provision?
## Usage
Examples of how to use this template:
## Components
Use this section to provide a detailed review of the core components of your template.
* Describe the 'resources' that are deployed using this template
* Describe the dependencies between resources
* Describe the default configuration of the resources
* Describe any special policy considerations (UpdatePolicy, DeletePolicy, etc)
## Structure
CloudFormation templates describe your AWS resources and their properties.
Use this section to describe the structure of your Service Template:
/stack # Stack Template Root.
/parameters # Parameters Directory for the CloudFormation Template
default.json # Default Parameters to be used when Environment Specific Parameters File Does Not Exist
# Environment Specific Parameter Files can be implemented via dev.json, tst.json, prd.json
/policy # Policy Directory for the CloudFormation Stack Policy
default.json # Default Stack Policy to be used when Environment Specific Policy File Does Not Exist
# Environment Specific Policy Files can be implemented via dev.json, tst.json, prd.json
service.yaml # Service Template Configuration File
template.yaml # CloudFormation Template for the Service
README.md # README File describing the service template
## Issues
Describe how users should report any issues that are identified while using this Service Template
Example:
If you identify an issue using this template please contact support@example.com
Detailed information is very helpful to understand and prioritize issues.
How to reproduce the issue, step-by-step.
The expected behavior (or what is wrong).
The error message(s)
The operating system
Screenshots if applicable
## Contributions
Pull Requests (PRs) are always welcome. Please create a new branch and submit a pull-request for review.
## Questions
Still have questions? Please contact support@example.com
Stack Template
Here is an example stack template.yaml file:
---
# CloudFormation Stack Template
AWSTemplateFormatVersion: "2010-09-09"
Description: >
Here is a description and
some additional details
about this template.
Metadata:
template metadata
Parameters:
pName:
Description: Sets the Name. Default is <blank>
Type: String
Default: <blank>
pStack:
Description: Sets the Stack. Default is <blank>
Type: String
Default: <blank>
pWorkspace:
Description: Sets the Workspace. Default is dev.
Type: String
Default: dev
ConstraintDescription: Must be an allowed value. (dev|tst|stg|prd|ops)
AllowedValues:
- dev # Development
- tst # Testing
- stg # Staging
- prd # Production
- ops # Operations
Mappings:
set of mappings
Conditions:
set of conditions
Transform:
set of transforms
Resources:
set of resources
Outputs:
set of outputs
Workflow
Here is a sample Cloudformation Wrapper script for a more programmatic workflow, that uses JQ to parse the JSON parameter and policy files. The Cloudformation wrapper script performs the following basic actions:
- List Stack
- List Parameters
- Describe Stack
- Describe Stacks
- Deploy Stack
- Destroy Stack
- Stage Stack
- Validate Template
#!/usr/bin/env bash
## ARGUMENTS
## ====================================
ACTION=$1
STACK=$2
WORKSPACE=$3
if [ -z "$WORKSPACE" ]
then
echo "Workspace argument not specified, using 'default'"
WORKSPACE="default"
fi
## FUNCTIONS
## ====================================
function help () {
echo "Description:"
echo " CloudFormation Script is a simple programmatic workflow"
echo ""
echo "Usage: "
echo " ./cloudformation.sh <ACTION> <STACK> <WORKSPACE>"
echo ""
echo "Example:"
echo " ./cloudformation.sh list-stacks"
echo " ./cloudformation.sh list-stack s3-bucket"
echo " ./cloudformation.sh list-parameters s3-bucket dev"
echo " ./cloudformation.sh deploy-stack s3-bucket dev"
echo " ./cloudformation.sh describe-stack s3-bucket"
echo " ./cloudformation.sh describe-stacks"
echo " ./cloudformation.sh destroy-stack s3-bucket dev"
echo " ./cloudformation.sh stage-stack s3-bucket dev"
echo " ./cloudformation.sh validate-template s3-bucket"
echo ""
}
function describeStack () {
echo ""
echo "CloudFormation Script is Describing Stack '$STACK' in $WORKSPACE"
COMMAND="aws cloudformation describe-stacks --stack-name $STACK "
echo $COMMAND
eval $COMMAND
}
function describeStacks () {
echo ""
echo "CloudFormation Script is Describing Stacks"
COMMAND="aws cloudformation describe-stacks"
echo $COMMAND
eval $COMMAND
}
function stageStack () {
echo ""
echo "CloudFormation Script is getting Parameters for $STACK in $WORKSPACE "
PARAMS=$(jq -r '.[] | [.ParameterKey, .ParameterValue] | "\(.[0])=\"\(.[1])\""' ./stacks/$STACK/params/$WORKSPACE.json)
echo "-------------------------------------"
echo "./stacks/$STACK/params/$WORKSPACE.json"
echo "-------------------------------------"
echo "$PARAMS"
echo ""
echo "CloudFormation Script is Staging Stack '$STACK' in $WORKSPACE "
COMMAND="aws cloudformation deploy --stack-name $STACK --template-file ./stacks/$STACK/template.yaml --no-execute-changeset --parameter-overrides $PARAMS"
echo $COMMAND
echo ""
eval $COMMAND
}
function deployStack () {
echo ""
echo "CloudFormation Script is getting Parameters for $STACK in $WORKSPACE "
PARAMS=$(jq -r '.[] | [.ParameterKey, .ParameterValue] | "\(.[0])=\"\(.[1])\""' ./stacks/$STACK/params/$WORKSPACE.json)
echo "-------------------------------------"
echo "./stacks/$STACK/params/$WORKSPACE.json"
echo "-------------------------------------"
echo "$PARAMS"
echo ""
echo "CloudFormation Script is Deploying Stack '$STACK' in $WORKSPACE "
COMMAND="aws cloudformation deploy --stack-name $STACK --template-file ./stacks/$STACK/template.yaml --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --parameter-overrides $PARAMS"
echo $COMMAND
echo ""
eval $COMMAND
}
function destroyStack () {
echo ""
echo "CloudFormation Script is Destroying Stack '$STACK' in $WORKSPACE"
COMMAND="aws cloudformation delete-stack --stack-name $STACK "
echo $COMMAND
eval $COMMAND
}
function describeStacks () {
echo ""
echo "CloudFormation Script is Describing Stacks"
COMMAND="aws cloudformation describe-stacks"
echo $COMMAND
eval $COMMAND
}
function describeStack () {
echo ""
echo "CloudFormation Script is Describing Stack '$STACK'"
COMMAND="aws cloudformation describe-stack --stack-name $STACK"
echo $COMMAND
eval $COMMAND
}
function listParameters () {
echo ""
echo "CloudFormation Script is Listing Parameters for Stack '$STACK' in $WORKSPACE"
PARAMS=$(jq -r '.[] | [.ParameterKey, .ParameterValue] | "\(.[0])=\"\(.[1])\""' ./stacks/$STACK/params/$WORKSPACE.json)
echo "-------------------------------------"
echo "./stacks/$STACK/params/$WORKSPACE.json"
echo "-------------------------------------"
echo $PARAMS
echo ""
}
function validateTemplate () {
echo ""
echo "CloudFormation Script is Validating Template for Stack '$STACK'"
COMMAND="aws cloudformation validate-template --template-body file://./stacks/$STACK/template.yaml"
echo $COMMAND
eval $COMMAND
}
case "$ACTION" in
stage-stack)
stageStack
;;
deploy-stack)
deployStack
;;
describe-stack)
describeStack
;;
describe-stacks)
describeStacks
;;
destroy-stack)
destroyStack
;;
list-stack)
describeStack
;;
list-stacks)
describeStacks
;;
list-parameters)
listParameters
;;
validate-template)
validateTemplate
;;
*)
help
;;
esac
Next
The above structure and workflow are useful for initializing new cloudformation projects and maintaining them over time as they scale. The workflow via the cloudformation.sh wrapper is beneficial because it standardizes and reinforces a consistent structure. As an architect and consultant I am often collaborating with different customers and teams. This approach is especially helpful when working across multiple projects or rapidly prototyping new solutions
After using this basic pattern in practice for several AWS Cloud projects I have designed and developed an approach for building a “Pattern Library” as a mechanism for scaling knowledge across projects and teams. I am working on an open source project called Supply that can be used to implement similar patterns and approaches. More to come!
Resources
Let's Work Together
Let's Get Ship Done!
Handcrafted by a @KnownTraveler