Keeping your secrets out of Ansible Playbooks

This post was originally published on July 15, 2020, but the content has since been updated to adhere to latest best practices.

Every good web service needs a good authentication layer. AWS uses IAM to manage users, one of which is your development user (you’re not using the root account, are you?). How do we authenticate with AWS when Ansible? Let’s find out!

But first, important things about credentials

Credentials must be kept secret. Even AWS’s secret access key says so in its name - you mustn’t tell that to anyone. This can be a challenge with tools that you need to trust to keep your passwords a secret, but don’t worry, our AWS Collection will never disclose your credentials inadvertently.

The other extremely important thing about credentials is that you need to keep them secret. Yes, you need to keep them secret. They need to be kept out of source code repositories, code, and any other files that are not exclusively meant to be secrets. A quick and dirty “I’ll just put these credentials here directly and come back later to extract them.” can just as quickly lead to splitting migraines when GitHub scans your public repository and find one of your production keys, whereupon AWS revokes them, but not before someone else made out with a few free hours of m5dn.24xlarge runtime.

For flexibility, we offer all possible options for applying credentials to your Ansible Playbooks. It is your job to be safe. Now, let’s move on to the three main ways you can communicate AWS credentials to the modules.

The following methods are standard and can be used with other collections, even for other clouds. We are using the Steampunk AWS collection in the examples, but the core ideas are independent of the chosen modules.

Storing credentials in the Ansible Playbook directly, or with variables

I know we just said you must not input or store credentials directly in the Ansible Playbook, but let’s get that (very, very bad) baseline out of the way.

- hosts: localhost
  tasks:
    - steampunk.aws.ec2_vpc_info:
        auth:
          aws_access_key: my-access-key
          aws_secret_key: my-secret-key
          aws_ec2_region: eu-north-1
      register: vpcs

    - ansible.builtin.debug:
        var: vpcs

See how simple this is? Even though it’s the simplest way, it’s the worst. Let’s improve it.

- hosts: localhost
  tasks:
    - steampunk.aws.ec2_vpc_info:
        auth:
          aws_access_key: "{{ my_access_key }}"
          aws_secret_key: "{{ my_secret_key }}"
          aws_ec2_region: "{{ my_region }}"
      register: vpcs

    - ansible.builtin.debug:
        var: vpcs

For the above example to work, we need to provide variables to Ansible. The simplest way is creating a YAML file that contains them. Let’s call that file secret-vars.yml:

my_access_key: my-access-key
my_secret_key: my-secret-key
my_region: eu-north-1

To glue the two together, we use the following command:

$ ansible-playbook -e @secret-vars.yml playbook.yml

See how we’ve now easily and cleanly separated the secrets from the implementation. This is now already a very good way of not storing credentials, provided you gitignore or otherwise avoid sharing secret-vars.yml. If you’re here for only a simple, safe way of storing credentials, this is it. If you’d like to see more options, read on.

Variables in Ansible Vault

Ansible Vault is a way of password-protecting your credentials at rest. You create a vault (a keychain, password storage, etc) that stores all of your credentials, but you can only access them if you have the correct password. Let’s make one now, interactively:

$ ansible-vault create secret-vars.vault
# input a password
# input the password again
# paste the contents of secret-vars.yml from above into the opened editor
# save and exit

You now have a file named secret-vars.vault, which is, under the hood, an encrypted YAML file. It’s very, very similar to secret-vars.yml, has the same contents, and behaves almost exactly like it, but is encrypted with the password you’ve chosen. Its usage is also identical to that of variable files’, try it out:

ansible-playbook -e @secret-vars.vault playbook.yml

Ansible detects you’re using a Vault and prompts you for the password automatically. But we’re still using variables to configure the authentication for our module. What if we want to use multiple modules?

- hosts: localhost
  tasks:
    - steampunk.aws.ec2_vpc:
        name: my-new-vpc
        auth:
          aws_access_key: "{{ my_access_key }}"
          aws_secret_key: "{{ my_secret_key }}"
          aws_ec2_region: "{{ my_region }}"

    - steampunk.aws.ec2_vpc_info:
        auth:
          aws_access_key: "{{ my_access_key }}"
          aws_secret_key: "{{ my_secret_key }}"
          aws_ec2_region: "{{ my_region }}"

    - steampunk.aws.ec2_key_pair:
        name: my-new-keypair
        auth:
          aws_access_key: "{{ my_access_key }}"
          aws_secret_key: "{{ my_secret_key }}"
          aws_ec2_region: "{{ my_region }}"
      register: keypair

    - ansible.builtin.debug:
        var: keypair

This quickly gets tedious. But there’s a solution!

Environment variables

Environment variables are a very common way of storing configuration. That’s what credentials are - as opposed to code, and it’s helpful to store them separately. Look at the following example:

- hosts: localhost
  environment:
    AWS_ACCESS_KEY: "{{ my_access_key }}"
    AWS_SECRET_KEY: "{{ my_secret_key }}"
    AWS_EC2_REGION: "{{ my_region }}"
  tasks:
    - steampunk.aws.ec2_vpc:
        name: my-new-vpc

    - steampunk.aws.ec2_vpc_info:
        auth:

    - steampunk.aws.ec2_key_pair:
        name: my-new-keypair
      register: keypair

    - ansible.builtin.debug:
        var: keypair

Here, we set environment variables for the whole play, but source them from variables, just like before. This solves the problem of setting authentication parameters for each task separately, as it magically works due to the values being sourced from the environment. But it isn’t ideal. We still need to invoke the damn thing with

$ ansible-playbook -e @secret-vars.yml playbook.yml

which still requires us to create, and maybe mistakenly publish, a new file. Here’s a solution: don’t use variables. We can just pass environment variables through to tasks directly without referencing them anywhere:

- hosts: localhost
  tasks:
    - steampunk.aws.ec2_vpc:
        name: my-new-vpc

    - steampunk.aws.ec2_vpc_info:
        auth:

    - steampunk.aws.ec2_key_pair:
        name: my-new-keypair
      register: keypair

    - ansible.builtin.debug:
        var: keypair

This is the cleanest option. It’s up to us to set the appropriate environment variables and, because that’s just about the easiest and most flexible method of configuration, we can just

$ AWS_ACCESS_KEY=abc AWS_SECRET_KEY=xyz AWS_EC2_REGION=eu-north-1 \
    ansible-playbook playbook.yml

While the above is longer than previous invocations of ansible-playbook, it’s also (arguably) the safest and (definitely) the most powerful. You can decide on your preferred method of authentication and your credentials won’t be stored anywhere, as they’re not present in any file.

To see the benefits of this flexibility however, we need to explore yet another way of authenticating with AWS.

AWS CLI configuration and profiles

Did you know AWS has an official CLI client ? Well, one of its major features is the ability to have a persistent login. Typing aws configure allows you to set access credentials for your user and store them on the local machine, in your home directory, away from your code. Client applications, such as boto3 , the official Python SDK for AWS, can then use those credentials automatically. No need to worry about them being set when you execute your Ansible Playbook.

A very interesting feature of this credential storage are profiles. These are sets of settings and credentials - you can have a profile named personal and a profile named work, and switch between them with a single variable. Let’s create the work profile now.

The following uses non-interactive configuration. Use aws configure --profile work to be interactively prompted for all this information.

$ aws configure set --profile work aws_access_key_id     my-access-key
$ aws configure set --profile work aws_secret_access_key my-secret-key
$ aws configure set --profile work region                eu-north-1

After this, we can run the Ansible Playbook we used before with slightly different authentication parameters:

$ AWS_PROFILE=work ansible-playbook playbook.yml

Isn’t this nice! All of this flexibility is fully supported by our Steampunk AWS Ansible collection.

You may have noticed that we’re running everything on localhost. Well, here’s why.

An advanced example: authentication jump hosts

The place our tasks get executed in all examples above is the local machine, even though we’re configuring services on AWS. This is also the place our profiles and credentials are commonly stored. However, you might have a need to use a jump host to connect to AWS’s APIs. A jump host is a machine you have access to, which has access to credentials (or the network) of the thing you want to connect to. It makes much more sense when managing machines, not APIs, but it is still a very valid configuration.

The Ansible Playbook we’ve been using above still remains the same (isn’t this nice?), except for the host we’re running our tasks on. It was localhost, but let’s change it to my-jumphost. Executing the playbook can be done with, as a theoretical example:

$ AWS_PROFILE=jumphost-eb7afc09 ansible-playbook playbook.yml

This uses a profile on the remote machine, the jump host. How neat!

Other options

But to finish things off, let’s look at something less enterprisey: using a password manager. We’ll use OnePassword as an example (other providers are available).

- hosts: localhost
  environment:
    AWS_ACCESS_KEY: "{{ lookup('onepassword', 'my_access_key_in_onepassword')  }}"
    AWS_SECRET_KEY: "{{ lookup('onepassword', 'my_secret_key_in_onepassword')  }}"
    AWS_EC2_REGION: eu-north-1
  tasks:
    - steampunk.aws.ec2_vpc:
        name: my-new-vpc

    - steampunk.aws.ec2_vpc_info:
        auth:

    - steampunk.aws.ec2_key_pair:
        name: my-new-keypair
      register: keypair

    - ansible.builtin.debug:
        var: keypair

Things will just work! What we used is called a lookup plugin, a less-than-optimally named feature of Ansible that looks up a value, in this case from OnePassword. If you’re logged in, you don’t need to worry about anything, your Ansible Playbooks will be secure, and you’ll have one less thing to worry about.

Wrapping up

If you take anything away from this post, it should be that you must not store your credentials in your playbooks. We’d be delighted if you tried the various authentication schemes yourself.

This is just one step to improving the quality of you playbooks. If you’d like to learn more about making high-quality Ansible content, check out Steampunk Spotter , Ansible Playbooks Platform that helps you make your playbooks more reliable, secure, and can even assist you in upgrading your playbooks to a newer version.