Managing remotely-generated API keys with Ansible

Question

I'm using ansible to provision a particular service, and before I can interact with it I must first generate an API key. But I can't predefine that key in my playbook (as a secret) - it is generated by the server, returned to me once, and will never be exposed again (the typical scnenario for API keys).

I could ask the server to generate a new API key every time I run that playbook, or tags in that playbook, but that is very slow. Ideally I should save that API key locally and reuse it.

My approach has been to simply write it to a local file in my user directory (~/.ansible/custom/api_key.txt) so it has some protection. That works but feels kind of dirty.

Does ansible have an official / robust way to handle this scenario?

Answer 1

This is kind of a broad question. There are a variety of ways of handling secrets in your playbooks; this article describes several options, and there are a variety of articles online that cover similar topics.

A simple option might be to encrypt the API key to a GPG public key for which the private key is only available when you're logged in and able to provide the passphrase.

Here's a simple (i.e. not particularly robust) example:

- hosts: localhost
  gather_facts: false

  tasks:
    # Check if the file in which we cache the API key exists.
    # If not, fetch the API key from the API and store it in
    # a GPG-encrypted file.
    - when: apikey_file is not file
      block:
        # This is just a dummy task to give us a string; you would of
        # course replace this with the logic to acquire an API key.
        - name: Get key from API
          command: echo secret.key
          register: apikey

        # Encrypt the key using the public key for the identity
        # stored in apikey_gpg_id.
        - name: Write API key to file
          command: >-
            gpg -o "{{ apikey_file }}" -e -r "{{ apikey_gpg_id }}"
          args:
            stdin: "{{ apikey.stdout }}"

- hosts: localhost
  gather_facts: false

  tasks:
    # Decrypt the API key file to stdout. This requires us to type in
    # the passphrase (which may be cached for some amount of time in your
    # GPG agent).
    - name: Read API key from file
      command: >-
        gpg -d "{{ apikey_file }}"
      register: apikey

    # Show what we got from the previous task.
    - debug:
        var: apikey.stdout

This presumes that the variables apikey_file and apikey_gpg_id are defined in advance -- I've put them in group_vars/all.yaml, but they could also be defined in your inventory or elsewhere depending on how your project is structured.

Answer 2

Use passwordstore. Ansible provides a lookup plugin. See

shell> ansible-doc -t lookup passwordstore

Install, initialize passwordstore in the current directory for testing, and export path

shell> export PASSWORD_STORE_DIR=$PWD/.password-store

Show content. The passwordstore is empty

shell> pass
Password Store

Create a project for testing

shell> tree -a .
.
├── ansible.cfg
├── hosts
├── .password-store
│   └── .gpg-id
└── pb.yml

shell> cat ansible.cfg
[defaults]
gathering = explicit
collections_path = $HOME/.local/lib/python3.9/site-packages/
inventory = $PWD/hosts
roles_path = $PWD/roles
remote_tmp = ~/.ansible/tmp
retry_files_enabled = false
stdout_callback = yaml

Create inventory

shell> cat hosts
host_A
host_B
host_C

In the first block test whether apikey(s) are missing or empty

- hosts: all

  vars:

    project: test-400
    passwd_dict: "{{ dict(passwd_out.results|
                          json_query('[].[item, ansible_facts.dummy]')) }}"

  tasks:

    - name: Test apikey is empty or missing
      block:

        - set_fact:
            dummy:  "{{ lookup('community.general.passwordstore',
                                entity,
                                missing='empty') }}"
          loop: "{{ ansible_play_hosts_all }}"
          loop_control:
            label: "{{ entity }}"
          register: passwd_out
          vars:
            entity: "{{ project }}/{{ item }}/apikey"

        - debug:
            var: passwd_out
          when: debug_classified|d(false)|bool

        - debug:
            var: passwd_dict
          when: debug_classified|d(false)|bool

      run_once: true

Because the passwordstore is empty we get

  passwd_dict:
    host_A: ''
    host_B: ''
    host_C: ''

In the second block get and store apikey(s) if empty or missing. Change the method of getting the apikey to your needs

    - name: Get and store apikey if empty or missing
      block:
     
        - name: Get apikey
          set_fact:
            apikey: "{{ lookup('password', '/dev/null', seed=inventory_hostname) }}"

        - debug:
            var: apikey
          when: debug_classified|d(false)|bool

        - name: Store apikey
          set_fact:
            dummy: "{{ lookup('community.general.passwordstore',
                               entity,
                               create=true,
                               userpass=apikey) }}"
          vars:
            entity: "{{ project }}/{{ inventory_hostname }}/apikey"

      when: passwd_dict[inventory_hostname]|length == 0

gives the created apikey(s)

TASK [debug] **********************************************************************************
ok: [host_B] => 
  apikey: YlOTwY9jviKhVaxokzbj
ok: [host_A] => 
  apikey: szHcyJNh-vnU-XXsuWt-
ok: [host_C] => 
  apikey: 67x4AcAK6_6liU1Ji,8u

The passname(s) were stored in passwordstore

shell> pass
Password Store
└── test-400
    ├── host_A
    │   └── apikey
    ├── host_B
    │   └── apikey
    └── host_C
        └── apikey

shell> pass test-400/host_A/apikey 
szHcyJNh-vnU-XXsuWt-
lookup_pass: First generated by ansible on 26/06/2023 12:34:59

shell> pass test-400/host_B/apikey 
YlOTwY9jviKhVaxokzbj
lookup_pass: First generated by ansible on 26/06/2023 12:34:59

shell> pass test-400/host_C/apikey 
67x4AcAK6_6liU1Ji,8u
lookup_pass: First generated by ansible on 26/06/2023 12:34:59

In the play, you can limit the scope of the apikey(s) to a task. For example,

    - name: Limit scope of apikey(s) to task
      block:
        - set_fact:
            dummy: ''
            passwd_dict: {}
        - debug:
            msg: "Use {{ entity }}: {{ apikey }}"
          vars:
            entity: "{{ project }}/{{ inventory_hostname }}/apikey"
            apikey: "{{ lookup('community.general.passwordstore', entity) }}"
      when: scope|d('play') == 'task'

gives

TASK [debug] **********************************************************************************
ok: [host_A] => 
  msg: 'Use test-400/host_A/apikey: szHcyJNh-vnU-XXsuWt-'
ok: [host_C] => 
  msg: 'Use test-400/host_C/apikey: 67x4AcAK6_6liU1Ji,8u'
ok: [host_B] => 
  msg: 'Use test-400/host_B/apikey: YlOTwY9jviKhVaxokzbj'

Otherwise, use the dictionary passwd_dict when you decide the play scope is fine. The below block gives the same result

    - name: No limit of apikey(s)
      block:
        - set_fact:
            dummy: ''
        - debug:
            msg: "Use {{ entity }}: {{ passwd_dict[inventory_hostname] }}"
          vars:
            entity: "{{ project }}/{{ inventory_hostname }}/apikey"
      when: scope|d('play') == 'play'

---

Example of a complete playbook for testing

- hosts: all

  vars:

    project: test-400
    passwd_dict: "{{ dict(passwd_out.results|
                          json_query('[].[item, ansible_facts.dummy]')) }}"

  tasks:

    - name: Test apikey is empty or missing
      block:

        - set_fact:
            dummy:  "{{ lookup('community.general.passwordstore',
                                entity,
                                missing='empty') }}"
          loop: "{{ ansible_play_hosts_all }}"
          loop_control:
            label: "{{ entity }}"
          register: passwd_out
          vars:
            entity: "{{ project }}/{{ item }}/apikey"

        - debug:
            var: passwd_out
          when: debug_classified|d(false)|bool

        - debug:
            var: passwd_dict
          when: debug_classified|d(false)|bool

      run_once: true

    - name: Get and store apikey if empty or missing
      block:
     
        - name: Get apikey
          set_fact:
            apikey: "{{ lookup('password', '/dev/null', seed=inventory_hostname) }}"

        - debug:
            var: apikey
          when: debug_classified|d(false)|bool

        - name: Store apikey
          set_fact:
            dummy: "{{ lookup('community.general.passwordstore',
                               entity,
                               create=true,
                               userpass=apikey) }}"
          vars:
            entity: "{{ project }}/{{ inventory_hostname }}/apikey"

      when: passwd_dict[inventory_hostname]|length == 0

    - name: Limit scope of apikey(s) to task
      block:
        - set_fact:
            dummy: ''
            passwd_dict: {}
        - debug:
            msg: "Use {{ entity }}: {{ apikey }}"
          vars:
            entity: "{{ project }}/{{ inventory_hostname }}/apikey"
            apikey: "{{ lookup('community.general.passwordstore', entity) }}"
      when: scope|d('play') == 'task'

    - name: No limit of apikey(s)
      block:
        - set_fact:
            dummy: ''
        - debug:
            msg: "Use {{ entity }}: {{ passwd_dict[inventory_hostname] }}"
          vars:
            entity: "{{ project }}/{{ inventory_hostname }}/apikey"
      when: scope|d('play') == 'play'