One of the first things I learned as a junior developer was to never push secrets to git. Sharing the secrets needed to bootstrap an environment has always been a bit wonky and I often wonder if we are not creating bigger security holes than what we were trying to avoid with not having secrets in a (private) repo. Another side effect of not having a proper way to handle secrets is that these things do not get rotated as often as they should have.
It is important to note that tools like HashiCorp Vault address a lot of the problems associated with sharing and distributing secrets, but even with Vault, you still need to bootstrap its trust at some point.
This problem got much more complicated when I started with infrastructure as code especially when doing it across many AWS accounts and projects. Keeping track of all these secrets became a nightmare, I really just wanted a way to check these secrets into git.
Having your secrets in the public domain is actually quite common. If you’ve ever used a service like Travis CI, then you’ll know that we store our secrets as secure environment variables in code. The trick here is that there is a central service (Travis) that can decrypt these secrets when they are needed, so how do we do this for something like Terraform, where you can do a deployment from your local machine or centrally using some CD tool (Terraform Cloud, GoCD, Github Actions, …).
👮♀️ sops to the rescue
No surprise that it is pretty common for developers to want to put secrets in code. The engineering team at Mozilla built sops for exactly this purpose, sops allow you to encrypt the keys in a key-value pair for commonly used file formats like JSON and YAML. What makes it really awesome is that it supports a wide array of cryptographic service providers like PGP, AWS KMS, GCP KMS and Azure Key Vault.
Imagine I have a file with the following content I want to use in my Terraform (TF) configuration:
I can encrypt this file with sops using AWS KMS with the following command:
I now have a file called sops.enc.yaml that I can push to git and only people or systems with sufficient permissions on the KMS key can decrypt the content. Below is the content of the encrypted file. Note how only the keys of key-value pairs are encrypted. This allows you to do a proper git diff on this secret file to track changes to it, the downside is you are potentially leaking a bit of metadata about what the secrets are used, if this is a concern you can encrypt the whole file, but loose some of the benefits of having it in git.
Also, notice the aws_profile="" in the file, this means I used my default AWS credentials in my environment to access the KMS key. You can change this to any profile you have configured in your AWS CLI config or credentials file. Additionally, you can specify a role in the KMS block as described in the official sops docs.
This solves storing the secrets in code, but how do use this in TF. The simplest way would be for team members to decrypt the file and then load it up in TF using something like this.
First, decrypt the file:
Load the secrets into TF. In TF 0.15 and later you can mark the value as sensitive that will prevent it from being displayed during plan and apply stages:
Although this will work, it’s a tedious workflow and you stand the chance of the unencrypted file landing up in code, additionally, it will not work if you are using something like Terraform Cloud to manage your deployments.
terraform-provider-sops
sops contain a go package called decrypt which can be used by other go programs to decrypt sops files. Calle Pettersson used this package to build terraform-provider-sops, this TF provider allows you to load encrypted files directly into TF without the need to decrypt them first.
A simple example of how this can be used:
It is important to note that when trying to decrypt the secrets sops will need access to whatever cryptographic service provider you used, for example, if you used AWS KMS, sops used the Go AWS SDK, which will look for credentials to use in the local environment where this is run.
If you have other ways to sharing secrets in code or with team members, I’d love to hear about it.
Comments