Node connects to chef server, workstation connects to chef server, but bootstrapping same node fails

Question

I have 3 components: node, server and workstation. I did not set these up. I can run a knife status from the workstation and it returns current results (I believe this confirms connection from workstation to server). I can successfully run chef-client from the node itself (I believe this confirms connection from node to server + clearly that it has already been bootstrapped).

But trying to bootstrap again fails. One thing I find to be odd is that I noticed the /etc/chef/client.rb file on the workstation gets its node_name line replaced with the name of the node that I'm trying to bootstrap. Maybe that's normal?

One thing I can't sort out is that this error makes it look like /someone/ can't connect to the chef server; however, as stated above, I confirmed both workstation and node can auth and connect to the chef server.

I've been looking through the documentation, but a huge problem is the ambiguity. It will say "some thing is located in /etc/chef/client.rb" - ok, but on the node, server or workstation? They all have this file!!

I think it may be my key which is indicated in the ~/.chef/knife.rb on the workstation by the "client_key" line - but I don't want to change that, lest I lose connection to the server. But it seems clearly like there is an authentication breakdown somewhere - I just can't find it anywhere but during bootstrap.

[root@workstation chef]# cat client.rb
log_location     STDOUT
chef_server_url  "https://chefserver/organizations/domain"
validation_client_name "domain-validator"
node_name "workstation"
ssl_verify_mode :verify_none
trusted_certs_dir "/etc/chef/trusted_certs"

[root@workstation chef]# knife bootstrap node.domain.goes.here -N node -x user -i /home/user/.ssh/id_rsa --sudo --use-sudo-password --node-ssl-verify-mode none -r recipe[chef-client]
Doing old-style registration with the validation key at ~/.keys/domain-validator.pem...
Delete your validation key in order to use your user credentials instead

Connecting to node.domain.goes.here
node.domain.goes.here -----> Existing Chef installation detected
node.domain.goes.here Starting the first Chef Client run...
node.domain.goes.here Starting Chef Client, version 12.10.24
node.domain.goes.here
node.domain.goes.here ================================================================================
node.domain.goes.here Chef encountered an error attempting to load the node data for "node"
node.domain.goes.here ================================================================================
node.domain.goes.here
node.domain.goes.here Authentication Error:
node.domain.goes.here ---------------------
node.domain.goes.here Failed to authenticate to the chef server (http 401).
node.domain.goes.here
node.domain.goes.here Server Response:
node.domain.goes.here ----------------
node.domain.goes.here Invalid signature for user or client 'node'
node.domain.goes.here
node.domain.goes.here Relevant Config Settings:
node.domain.goes.here -------------------------
node.domain.goes.here chef_server_url   "https://chefserver/organizations/domain"
node.domain.goes.here node_name         "node"
node.domain.goes.here client_key        "/etc/chef/client.pem"
node.domain.goes.here
node.domain.goes.here If these settings are correct, your client_key may be invalid, or
node.domain.goes.here you may have a chef user with the same client name as this node.
node.domain.goes.here
node.domain.goes.here Platform:
node.domain.goes.here ---------
node.domain.goes.here x86_64-linux
node.domain.goes.here
node.domain.goes.here
node.domain.goes.here Running handlers:
node.domain.goes.here [2016-12-13T13:51:26-05:00] ERROR: Running exception handlers
node.domain.goes.here Running handlers complete
node.domain.goes.here [2016-12-13T13:51:26-05:00] ERROR: Exception handlers complete
node.domain.goes.here Chef Client failed. 0 resources updated in 01 seconds
node.domain.goes.here [2016-12-13T13:51:26-05:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
node.domain.goes.here [2016-12-13T13:51:26-05:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
node.domain.goes.here [2016-12-13T13:51:26-05:00] ERROR: 401 "Unauthorized"
node.domain.goes.here [2016-12-13T13:51:26-05:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

[root@workstation chef]# cat client.rb
log_location     STDOUT
chef_server_url  "https:/chef/organizations/domain"
validation_client_name "domain-validator"
node_name "node"
ssl_verify_mode :verify_none
trusted_certs_dir "/etc/chef/trusted_certs"

Answer 1

You probably already have a client matching that name in your org. Remove it via knife client delete.

Answer 2

What I understand is that you're trying to bootstrap again with a new name and this can't work. The node has already been bootstraped and it already have a client.pem file for a client with its old name, let's say old-name.

Chef won't try to register a new client with name new_name (node in your question) because the client.pem is here. But it will use new_name, result is a client new_name is trying to authenticate with a key of someone else.

For the chef-server, new_name does not exist, so it can't authenticate it.

I'm unsure of why you want to bootstrap again, so two solutions:

1. You want to change the node name, in this case remove the client.pem file on the node and run your bootstrap command again.
2. You want to run chef again on the node, in this case use whatever you want, a test model can be to use knife ssh -x user -i /home/user/.ssh/id_rsa 'name:old_name' 'sudo chef-client -r 'new_runlist' for this.

To remove the client.pem, you can do it via knife ssh also.

To pass the password in your command line see this Q/A on superuser, take care the password will be visible in the process list and in your shell history if you don't preppend both commands with a space.

Answer 3

I faced similar issue while bootstrapping again. After debugging what I found was that when it (bootstrapping) was done first time it created a chef file in the etc folder of the node. I removed that folder and bootstrapped again. It worked for me.