Ansible GCP IAP tunnel

Question

I’m trying to connect to a GCP compute instance through IAP. I have a service account with permissions.

I have tried the following

1. Basic ansible ping,ansible -vvvv GCP -m ping, which errors because the host name is not found bc I do not have an external ip
2. I have set ssh_executeable=wrapper.sh like here

Number 2 is almost working but regexing commands are hacky.

Is there a native ansible solution?

Edit: The gcp_compute dynamic inventory does work for pinging instances but it does not work for managing the instances.

Ansible does NOT support package or system management while tunneling through IAP.

Answer 1

gce dynamic inventory does not work unless all the inventory are publicly accessible. For private ip, the tunnel is not invoked when ansible commands are executed. The gce dynamic inventory will return inventory, but you can't actually send commands if behind a tunnel and private IP only. The only work around i could find is to have the ssh binary point at a custom script which calls the gcloud wrapper.

Answer 2

For those who are still looking for a solution to use IAP SSH with Ansible on an internal IP. I've made some changes to the scripts listed here

My main problem was the fact that I had to add --zone as an option, as gcloud wouldn't automatically detect this when run through Ansible. As I didn't want to call the CLI, adding more waittime, I've opted for using group_vars to set my ssh options. This also allows me to specify other options to the gcloud compute ssh command.

Here are the contents of the files needed for setup:

ansible.cfg

[inventory]
enable_plugins = gcp_compute

[defaults]
inventory = misc/inventory.gcp.yml
interpreter_python = /usr/bin/python

[ssh_connection]
# Enabling pipelining reduces the number of SSH operations required
# to execute a module on the remote server.
# This can result in a significant performance improvement 
# when enabled.
pipelining = True
scp_if_ssh = False
ssh_executable = misc/gcp-ssh-wrapper.sh
ssh_args = None

misc/gcp-ssh-wrapper.sh

#!/bin/bash
# This is a wrapper script allowing to use GCP's IAP SSH option to connect
# to our servers.

# Ansible passes a large number of SSH parameters along with the hostname as the
# second to last argument and the command as the last. We will pop the last two
# arguments off of the list and then pass all of the other SSH flags through
# without modification:
host="${@: -2: 1}"
cmd="${@: -1: 1}"

# Unfortunately ansible has hardcoded ssh options, so we need to filter these out
# It's an ugly hack, but for now we'll only accept the options starting with '--'
declare -a opts
for ssh_arg in "${@: 1: $# -3}" ; do
        if [[ "${ssh_arg}" == --* ]] ; then
                opts+="${ssh_arg} "
        fi
done

exec gcloud compute ssh $opts "${host}" -- -C "${cmd}"

group_vars/all.yml

---
ansible_ssh_args: --tunnel-through-iap --zone={{ zone }} --no-user-output-enabled --quiet

As you can see, by using the ansible_ssh_args from the group_vars, we can now pass the zone as it's already known through the inventory.

If you also want to be able to copy files through gcloud commands, you can use the following configuration:

ansible.cfg

[ssh_connection]
# Enabling pipelining reduces the number of SSH operations required to
# execute a module on the remote server. This can result in a significant
# performance improvement when enabled.
pipelining = True
ssh_executable = misc/gcp-ssh-wrapper.sh
ssh_args = None
# Tell ansible to use SCP for file transfers when connection is set to SSH
scp_if_ssh = True
scp_executable = misc/gcp-scp-wrapper.sh

misc/gcp-scp-wrapper.sh

#!/bin/bash
# This is a wrapper script allowing to use GCP's IAP option to connect
# to our servers.

# Ansible passes a large number of SSH parameters along with the hostname as the
# second to last argument and the command as the last. We will pop the last two
# arguments off of the list and then pass all of the other SSH flags through
# without modification:
host="${@: -2: 1}"
cmd="${@: -1: 1}"

# Unfortunately ansible has hardcoded scp options, so we need to filter these out
# It's an ugly hack, but for now we'll only accept the options starting with '--'
declare -a opts
for scp_arg in "${@: 1: $# -3}" ; do
        if [[ "${scp_arg}" == --* ]] ; then
                opts+="${scp_arg} "
        fi
done

# Remove [] around our host, as gcloud scp doesn't understand this syntax
cmd=`echo "${cmd}" | tr -d []`

exec gcloud compute scp $opts "${host}" "${cmd}"

group_vars/all.yml

---
ansible_ssh_args: --tunnel-through-iap --zone={{ zone }} --no-user-output-enabled --quiet
ansible_scp_extra_args: --tunnel-through-iap --zone={{ zone }} --quiet

Answer 3

not a direct answer to the OP, but after having crushed my head on how to keep my project safe (via IAP) and let ansible work at reasonable speed, I've ended up with a mix of IAP and OS Login. This continues to use the dynamic inventory if needed.

I use IAP and no public IPs on my VMs, then I've enabled OS Login project wide and I've created a small "ansible-server" VM internal to the project (well this is a WIP as in the end a VPC paired project should CI/CD ansible but this is another story).

• Inside the VM I've setup the identity of a dedicated service account via

gcloud auth activate-service-account [email protected] --key-file=/path/to/sa/json/key

• then I've created a pair of ssh keys
• I've enabled the S.A. to login by exporting the public key via

gcloud compute os-login ssh-keys add --key-file ~/.ssh/my-sa-public-key

• I run all my playbooks from within the VM passing the -u switch to ansible. This is blazing fast and let me revoke any permission via IAM avoiding floating ssh keys abandoned into project or VM metadata.

So the flow now is:

• I use IAP to login from my workstation into the ansible VM inside the project
• I clone the ansible repo inside the VM
• I run ansible impersonating the S.A.

Caveats:

• to get the correct username to be passed to ansible (via -u) record the username provided by the previous os-login command (it appears in the output of the added key, in my case was somethins like sa_[0-9]*)
• be sure the S.A. has both Service Account User and OS Admin Login IAM roles or the ssh will fail
• of course this means you have to keep a VM inside the project dedicated to ansible and also that you need to clone the ansible code into the VM. In my case, I mitigate the "issue", just switch the VM on/off on demand and I use the same public key to grant read-only access to the ansible repo (in my case on bitbucket)

Answer 4

(Converting my comment as an answer as requested by OP)

Ansible has a native gce dynamic inventory plugin that you should use to connect to your instances.

Answer 5

To make lotjuh's answer work I had to also update my inventory.gcp.yml file to have the following

plugin: gcp_compute
projects:
  - myproject
auth_kind: application
hostnames:
  - name

Without the hostnames: - name I was getting gcloud ssh errors since it tried to ssh into the instances using their host IP.

This approach also requires that the project be set in the gcloud config with gcloud config set project myproject