How to Debug Jenkins Pipeline node hangs on allocating node

1
votes

Any suggestions on debugging a pipeline hang?

When Jenkins 2.7.2 with pipeline plugins 2.2 cannot execute any pipeline job attempting to allocate a node.

Test

  1. 20+ nodes free
  2. 3 free executors on master
  3. Start pipeline job
  4. got gets to allocating node and hangs
  5. start DoNothing freestyle job that does nothing but run on a node
  6. DoNothing freestyle waits in queue even though 20+ nodes report as free
  7. Kill pipeline job
  8. DoNothing freestyle job finds a node, executes, completes

Clues - alas, not many

There's nothing in jenkins master log and in my preprod with the same plugins, everything works.

I suspect I have something getting in the way but i'm not sure what and really don't want to restart (but I think I need to)

Logs

[Pipeline] node
Still waiting to schedule task
Waiting for next available executor

Pipeline Script

echo 'Starting'
sleep time: 1, unit: 'NANOSECONDS'
echo 'Slept 1 NS'

node {
    echo "On a node"
}
1
jenkinsjenkins-pipeline
Have you checked that you actually have available executors on that Jenkins? – izzekil
Thanks. I do have many executors free. I can start a tiny pipeline job then start a tiny non pipeline job and both hold until I kill the pipeline. – Peter Kahn

1 Answers

1
votes

Restarting jenkins cleared the problem. I suspect, when we attempted to clear a hung build by killing a thread, we selected the wrong thread creating the problem or it was just a fluke.