How do I get Puppet Master to run a manifest against a Puppet Agent node when there are different errors?

Question

I have set up a puppet master and puppet agent server in one network. I have signed the puppet agent's cert request from the puppet master server. I am trying to run a manifest from Puppet Master against the Puppet agent server, but it is failing. The newManifest.pp on both Puppet Master servers calls the Puppet Agent through its FQDN. The /etc/hosts file on both servers has the entries of the FQDN of both servers and their IP addresses. I can ping each server from the other. I can SSH from each server to the other.

All four of these servers in both networks have CentOS 7 as the OS with open source Puppet version 3.8.4. The firewalls are turned off on all four machines. These ports are not blocked on any intermediate path: 22, 443, 8140, and 61610 on either network between the Puppet Master and Puppet Agent servers.

In network one, I expect manifests to compile. They never have.

I tried this with sshd_config` to use ports 22, 443, 8140, and 61610 on both the Puppet Master and Puppet agent. When I use this from the puppet master:

puppet agent -t newManifest.pp --server='foobar.acme.com'

I get an error:

Unable to fetch my node definition, but the agent will continue: Warning: Connection refused - connect(2).

I tried again with sshdconfig to not use any ports specifically. I recycled the sshdconfig services. I run the same command:

puppet agent -t newManifest.pp --server='foobar.acme.com'

I get a different error:

Unable to fetch my node definition, but the agent run will continue: connection refused - connect(2).

I then tried this command (with no explicit server designation):

puppet agent -t newManifest.pp

This time, I get this error:

Unable to fetch my node definition, but the agent will continue: Network is unreachable.

In network two, I expect manifests to work. They never have.

I tried this with sshd_config to use ports 22, 443, 8140, and 61610 on both the Puppet Master and Puppet agent. When I use this from the puppet master:

puppet agent -t newManifest.pp --server='foobar.acme.com'

I get a different error:

   Unable to fetch my node definition, but the agent run will continue: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol.

I tried again with sshdconfig to not use any ports specifically. I recycled the sshdconfig services. I run the same command:

puppet agent -t newManifest.pp --server='foobar.acme.com'

I get a different error:

 Unable to fetch my node definition, but the agent run will continue: connection refused - connect(2).

I then tried this command (with no explicit server designation):

 puppet agent -t newManifest.pp

This time, I get this error:

 Unable to fetch my node definition, but the agent will continue: No route to host - connect(2).

The errors between network one and two are different except when sshd_config is not using any specific ports and the command to compile the manifest explicitly sets the FQDN. Why is this? What is wrong? I have tried to set up Puppet twice just to fix the first instance. I cannot completely reproduce the specific problems in the first network.

How do I get the Puppet Master server to compile a manifest? The puppet agents have signed certificates. I don't understand why the compilation of the manifests always fails.

Answer 1

You have at least two entirely separate groups of problems.

First, as @daxelrod commented, the Puppet agent just doesn't work like that. The point of master / agent mode is that the master decides what the node's configuration is supposed to be. Furthermore, the master does not serve manifest files to agents; rather, it evaluates whatever manifests and data are necessary for it to build a "catalog" of resources for the agent to apply. It is these catalogs that the master serves.

If you want the agent to apply only part of its configuration in any given run then you can use the --tags option to direct it to select a specific subset of the resources from its catalog. Tags available for selecting can be specified manually in your manifests, and Puppet automatically tags resources with class name segments based on where they are declared, but this does not break down cleanly along the lines of manifest files.

If you really want to apply specific manifest files to your machines then those manifest files and any data or resources they rely upon must be present on the target machine. In that case, you can use puppet apply instead of using the agent, as @daxelrod observed.

---

Second, as for your network problems, if indeed

The firewalls are turned off on all four machines. These ports are not blocked on any intermediate path: 22, 443, 8140, and 61610 on either network between the Puppet Master and Puppet Agent servers.

as you say, then about the only other plausible explanation I can see for "connection refused" failures such as the agent reports is that the puppetmaster is not actually running on the machine the agent attempts to connect to, at least not on the port the agent tries to connect on. That might be because of some kind of name resolution problem (i.e. "foobar.acme.com" resolves differently than you expect it to do), because the master is configured to listen on a different port than you think, or simply because you did not successfully start it.

The "Network is unreachable" appears simpler to diagnose: if you do not specify a server to the agent via the command line, and you do not configure one in the agent's configuration file, then Puppet tries to connect to a machine named "puppet". This is resolving for you to an address on an unreachable network. (The message would be different if the agent could not resolve the name at all.)

The really interesting error message is the the SSL handshaking error:

SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol.

in that case, the client has successfully connected and attempted to start an SSL handshake sequence, but it does not understand the response from the server. Most likely this means that the server is not speaking SSL back to the client at all, but I'm in no position to guess what it is speaking.

---

Additionally, you mention sshd_config several times in your question, as if that had something to do with the problem. Unless you're doing something very unusual, however, neither sshd nor any of its configuration files has any involvement whatever in Puppet runs.

---

Overall, I'm inclined to think that you are dealing with a combination of incorrect expectations, poor understanding of the main system and support components, and perhaps an overambitious start. Puppet has fairly good documentation, including for system installation. If you're installing from RPMs (maybe PuppetLabs's) then that will give you a head start, but you will still need to configure both the master and the agent(s) appropriately. I suspect you are falling down somewhere here.

By all means, do try to do this in smaller bites. One master and one agent, on the same network, is about as simple as you can go for a master / agent configuration. Get that working first.