Making Cloudformation less painful

Let's be honest, the developer experience for Cloudformation is pretty rough. It was originally released in 2010 well before infrastructure as code was the norm. Just the fact that it has survived for over a decade without changing the template format (2010-09-09 is still the latest!) is both impressive and the reason it is so painful to work with -- it was built for a different time.

🤕 Where It Hurts

Compared to more modern tools like Terraform and Pulumi, it's easy to call out the faults of Cloudformation.

• JSON/YAML files JSON and YAML are not programming languages and severely limit the ability to use basic programming constructs such as for loops and variables. In addition, they are brittle and error-prone when editing by hand.
• Typing/Code completion Writing templates involves a ton of flipping back and forth between the documentation and your editor. Seeing resource signatures in your editor is a huge performance boost.
• Feedback cycle Cloudformation is slow. Compared to other tools which can show you errors in your editor and quickly recover from a failed state, debugging/troubleshooting Cloudformation templates can be hugely time-consuming.
• Lagging resource support As of right now, aws-cloudformation/cloudformation-coverage-roadmap has almost 700 open issues dating back 3 years.

🙈 Should We Give Up?

So if Cloudformation is so rough, why do we use it? One of the tenets of AppPack is that it uses AWS-managed tooling wherever it is appropriate. For all the shortcomings of Cloudformation, it gives us exactly what we need to help users set up complex groups of resources in their AWS accounts.

We also want to minimize the tooling our users need to install. While today our CLI is required to spin up resources, we'd like to make all that functionality available in the browser as well (see this post for details). That's not too difficult when using the AWS API but rules out using third-party tools like Terraform without some significant gymnastics.

Staying "AWS native" has other benefits too. Newer tooling is constantly evolving. That means revising your configurations to make sure they maintain compatibility. The flip side of Cloudformation feeling dated is that templates require almost zero maintenance once created.

Finally, it makes AppPack more transparent. Users who want to look under the hood can poke around in the Cloudformation web console to see what resources were set up, the templates used, and their configuration parameters.

💊 Easing the Pain

We're not the only ones who have struggled with Cloudformation. A plethora of tooling has been built to help create and verify Cloudformation templates.

Cloudformation got you down?

You're not alone. Stop the madness and start your free 30-day trial of AppPack.

Get Started Now

Cloud Development Kit (CDK)

The most popular project right now is the CDK. It lets you write your templates using real programming languages with type checking. It includes pre-built "Constructs" to create complex groups of resources such as a VPC.

On the surface, it checks a lot of the boxes, but in practice, I found it painful to use. It improves on the syntax of Cloudformation, but documentation can be hard to come by once you need to do more complex tasks. Since it deviates from Cloudformation's API, you can't use the wealth of documentation around Cloudformation.

It feels like it is more focused on creating a dynamic Cloudformation stack template than using the traditional Parameters and Conditions to build up resources from a static template. This isn't a good fit for us because we can't expect our users to have cdk installed. While you can write CDK code in several languages, it requires the CLI which is written in Typescript. It might be possible to port this to the browser, but it's certainly not made for it.

awslabs/goformation

goformation looks like a great fit at first glance. It's written in Go (same as our CLI). It's typed, so it's easy to work with in the editor. It's a one-to-one with Cloudformation templates so there are plenty of docs and examples you can use.

Unfortunately, it has some not-so-edge cases which make it unusable in many scenarios. They've implemented strict typing in places where the Cloudformation templates are a little more flexible. A couple examples of issues we ran into were awslabs/goformation#467 and awslabs/goformation#304. The latter is linked to a pull request from AWS' CLI, copilot which resulted in them removing the library from the project in favor of a straight YAML conversion. It's not a great sign that folks inside AWS abandoned it to effectively write templates by hand.

aws-cloudformation/cfn-lint

cfn-lint can catch many errors in your templates without having to upload them to AWS for testing with Cloudformation. It's not perfect, but it is a huge timesaver. It can be run locally and as part of your CI pipeline to prevent silly mistakes from ending up in your final templates.

cloudtools/troposphere

troposphere lets you generate templates from Python code. Like goformation, it is a one-to-one match with Cloudformation, so our knowledge there transfers easily. Instead of building "Constructs" with the CDK, we can write Python functions to add common groups of resources to our templates (hello AWS::CloudFormation::CustomResource).

Unfortunately, it does not supply type definitions. That means development involves a lot of back-and-forth between the docs and the editor which hurts productivity.

MacHu-GWU/cottonformation-project

I found cottonformation just last week. It is a modern reimplementation of troposphere. Unlike its counterpart, it makes heavy use of typing and metadata which speeds up development considerably. The code is automatically generated from the AWS Cloudformation resource specification which should make it easy to keep up-to-date -- something troposhpere isn't always on top of.

The strike against this one is that it doesn't seem to be heavily used and is currently missing critical features such as Condition. I'll be keeping my eye on (and possibly contributing to) this one.

🥞 Our Current Stack

So with all that, we settled on a combination of troposphere and cfn-lint. It solves a lot of problems for us but isn't ideal.

• ☀️ We get a "real" programming language to write our templates.
• ⛈ We don't get true type hints in our editor. I'm hoping to resolve this with a switch to cottonformation in the future.
• 🌤 We get a tight feedback cycle with cfn-lint. It doesn't catch everything, so you still have to test in AWS. This is probably the best we can do here.
• 🌥 For resources that aren't supported by Cloudformation, we use Custom Resources. Again, probably the best we can do.

As you can see, we haven't made Cloudformation a joy to work with (I'm not sure any IaC tool fits that bill), but we have made it a little better with some external tooling. Do you have any recommendations for improving this stack? We'd love to hear your thoughts via Twitter/Reddit/carrier pigeon 🕊.