Deploying CFN with Boto3

Deploying CFN with Boto3

I have recently been involved with a project that required orchestrating the automating of various AWS Services to achieve the desired end output. To achieve this ‘Orchestration’ I decided to use AWS Step Machines as they easily wrap together and control the flow and data processing (State) between the tasks. The first step of this Step Function was to deploy an already existing Cloud Formation (CFN) template using Lambda and Python. A simple task that caused a bit of unnecessary pain due to the fact that the CFN Template was written in YAML, had lots of Intrinsic functions and the Cloud Formation API only accepts templates in JSON. This blog posts will cover off how I got around the parsing of this custom YAML into JSON and then submitted the payload to the API.

Template Example

The template it self does not matter so below is an example template that we will use. The following template deploys an S3 bucket and has 2 input parameters.

In my example this file resides within an S3 bucket and the file (key) called “SimpleS3.yml”.

Reading and deploying the Template

As the template resides within S3 and we need to provide it’s contents (as JSON) and our parameters to the Cloud Formation API, we need to do the following 4 things:

  • Stream the templates contents from our S3 Bucket
  • Parse this from YAML to JSON
  • Create a request body for the CFN API
  • Call the API to begin deploying our Stack

Now each section in more detail.

Stream the templates contents

We can create a stream using Boto3 and the S3 Client like this.

Now if we look at what resides within template_data we can see an Object called Body of type “botocore.response.Streaming” which, in my case has a length of 630 bytes.

Using the built in Python read method we can read this Octet stream into memory and do with it as we wish. Now, onto the parsing steps.

Parse this stream

As mentioned above, we can read this octet stream into memory using the read method and providing the entire length of the content to stream, as follows:

Now, this gives us our file contents as Bytes that we now need to parse from YAML to JSON. Here is where it gets fun!

One would assume you could use the ‘safe_load” method from the yaml library to parse this. So let’s try that and see what happens.

We get a lot of red and an exception thrown saying that the yaml library cannot parse the “!Ref” tag as it has no constructor telling it how to do so. I spent some time researching adding custom constructors to no avail. It was then that I stumbled upon a library called “cfn_flip” that was written to do exactly this.

Using the methods of this library we can easily parse our YAML to JSON using the to_json method and get no constructor errors.

Create Body and Post to API

Nice, now we can create our request body and hit the API.

This will trigger the Stack deployment and return the status of it.

If you wanted to wait for the Stack to complete before returning then consider using a waiter method. This was not applicable to me as the stack I am deploying prod takes around 45 minutes which is much longer than Lambda can run for.

I hope this blog was useful and showed a few methods of how we can easily utilise the CloudFormation API and write programs to do so.

See you soon 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *