Long running command in Ansible ending in failed status with host unreachable

3
votes

I have to run a command in Ansible, which takes approximately 30+ minutes to complete. The command has to be executed in serial. The command I am running is nodetool repair in Cassandra and, if we started it parallel, it will hand the process in all machines. As repair in Cassandra cannot run parallel in all machines.

So, we are running them in serial. However, the command sometimes takes long time to complete.

As the command is taking long and my Ansible playbook is dying after waiting for some time, with message node unreachable.

{"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}

Is there a way, that I can wait for the process to complete?

I am using serial: 1 for hosts and running below task

task:

- name: Execute nodetool repair
  command: {{cassandra_installation_dir}}/bin/nodetool repair -j 4
1
ansibleansible-playbook

1 Answers

4
votes

You should use async for this:

- name: Execute nodetool repair
  command: {{cassandra_installation_dir}}/bin/nodetool repair -j 4
  async: 3600
  poll: 10

This will run the command in asynchronous mode for max 3600 seconds (1h) and check if the command is finished every 10 seconds (which is default anyway). If the command doesn't finish after 1h, the task will fail.