chef-client connection errors after executing knife bootstrap

4
votes

I'm getting these errors [http://pastebin.com/BEgNh2mm][chef-client/server errors] after successfully bootstrapping a node with chef.

The initial error which occurs is as follows

(snippet from sever logs):

merb : chef-server (api) : worker (port 4000) ~ Started request handling: Wed Aug 08 19:04:32 +0200 2012 merb : chef-server (api) :worker (port 4000) ~ Params: {"name"=>"devhouse", "controller"=>"clients", "action"=>"create", "admin"=>false} merb : chef-server (api) : worker (port 4000) ~ Connection reset by peer - (Errno::ECONNRESET)

(snippet from client STDOUT)

[2012-08-08T19:47:57+02:00] INFO: * Chef 10.12.0 * [2012-08-08T19:47:57+02:00] INFO: Client key /etc/chef/client.pem is not present - registering [2012-08-08T19:47:58+02:00] INFO: HTTP Request Returned 409 Conflict: Client already exists [2012-08-08T19:47:58+02:00] INFO: HTTP Request Returned 403 Forbidden: You are not allowed to take this action. [2012-08-08T19:47:58+02:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out [2012-08-08T19:47:58+02:00] FATAL: Net::HTTPServerException: 403 "Forbidden"

From the above the error the only thing I can deduce is that the client failed to communicate with the server, but, the mind boggling thing is that, somehow the client was able to register with the server cause executing knife client list shows client which field in the list however updated attempt by client fails with 403 "Forbidden"

Chef common errors does not seem to have any records of such behavior.

4
chef-infra

4 Answers

10
votes

The problem had two parts to it:

  1. Connection reset by peer - (Errno::ECONNRESET)

  2. HTTP Request Returned 409 Conflict: Client already exists and FATAL: Net::HTTPServerException: 403 "Forbidden"

The first issue I was able to narrow down to chef not been able to communicate with rabbitmq, it seems somewhere along the line all settings on rabbitmq were removed.

The second issue, during my investigation (I believe this is probably documented somewhere) it became apparent that chef stores pem info in couchdb through rabbitmq before client generates it's client.pem file. Since that process was not completing due to issue 1, the client was created but authentication information was missing.

Solution

  1. stop all chef services (solr, server, webui) and remove all *.pem files on chef server located under /etc/chef/
  2. Ensure rabbitmq is configured with all the necessary info and update your chef server.rb file with information
  3. start-up chef services (solr, server, webui), this will cause all necessary .pem files to be generated again
  4. run knife client delete "node-which-had-issues"
  5. on new/bootstrapped node execute chef-client
1
votes

i was facing the same issue.

chef-client

[2013-11-05T14:21:32+05:30] INFO: *** Chef 0.10.4 ***
[2013-11-05T14:21:34+05:30] INFO: Client key /etc/chef/client.pem is not present - registering
[2013-11-05T14:21:36+05:30] INFO: HTTP Request Returned 500 Internal Server Error: Connection refused - connect(2)
[2013-11-05T14:21:36+05:30] ERROR: Server returned error for http://"chef-server":4000/clients, retrying 1/5 in 4s
[2013-11-05T14:21:40+05:30] INFO: HTTP Request Returned 409 Conflict: Client already exists
[2013-11-05T14:21:40+05:30] INFO: HTTP Request Returned 403 Forbidden: You are not allowed to take this action.
[2013-11-05T14:21:40+05:30] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-11-05T14:21:40+05:30] FATAL: Net::HTTPServerException: 403 "Forbidden"

I solved it by restarting the rabbitmq server on the machine where chef server was running.

0
votes

It sounds like chef-client is registering with chef-server with that hostname, then client.pem is being changed which will then cause a 403 when the cert doesn't pass the SSL check. Essentially chef-server has the public side of that cert so it can authenticate nodes.

If you are running a chef-client setup process twice, that would cause this.

0
votes

I had the Net::HTTPServerException: 403 "Forbidden" error and I ran the "knife acl" command and it fixed it.

Example:

knife acl bulk add group clients nodes NODE_NAME update,read

NODE_NAME = yourserver@yourdomain