1
votes

When running a cookbook with chef-client, the run fails with this error: EOFError: end of file reached

I'm fairly new to this, and I don't know what this means. The cookbook is a very simple wrapper around the community cookbook which should be good. https://supermarket.chef.io/cookbooks/powershell

Here's the error again:

Synchronizing Cookbooks:
  - windows (1.37.0)
  - ms_dotnet2 (0.0.0)
  - ms_dotnet45 (0.0.0)
  - powershell (3.1.0)
  - chef_handler (1.1.8)
  - ewe-powershell (0.1.0)

  ================================================================================
  Error Syncing Cookbooks:
  ================================================================================

  Unexpected Error:
  -----------------
  EOFError: end of file reached


  Running handlers:
[2015-11-18T08:30:15-08:00] ERROR: Running exception handlers
  Running handlers complete
[2015-11-18T08:30:15-08:00] ERROR: Exception handlers complete
  Chef Client failed. 0 resources updated in 24 seconds
[2015-11-18T08:30:15-08:00] FATAL: Stacktrace dumped to c:/chef/cache/chef-stacktrace.out
[2015-11-18T08:30:15-08:00] FATAL: EOFError: end of file reached

C:\WINDOWS\system32>

and here's the stacktrace:

Generated at 2015-11-18 08:30:15 -0800
EOFError: end of file reached
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/protocol.rb:153:in `read_nonblock'
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/protocol.rb:153:in `rbuf_fill'
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil'
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/protocol.rb:144:in `readline'
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line'
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new'
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/http.rb:1412:in `block in transport_request'
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/http.rb:1409:in `catch'
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/http.rb:1409:in `transport_request'
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/http.rb:1382:in `request'
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/http.rb:1375:in `block in request'
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/http.rb:852:in `start'
C:/opscode/chef/embedded/lib/ruby/2.0.0/net/http.rb:1373:in `request'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/http/basic_client.rb:65:in `request'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/http.rb:264:in `block in send_http_request'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/http.rb:298:in `block in retrying_http_errors'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/http.rb:296:in `loop'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/http.rb:296:in `retrying_http_errors'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/http.rb:260:in `send_http_request'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/http.rb:170:in `streaming_request'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/rest.rb:180:in `streaming_request'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/rest.rb:118:in `get'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/cookbook/synchronizer.rb:277:in `download_file'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/cookbook/synchronizer.rb:254:in `sync_file'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/cookbook/synchronizer.rb:152:in `block (2 levels) in sync_cookbooks'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/util/threaded_job_queue.rb:52:in `call'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/util/threaded_job_queue.rb:52:in `block (3 levels) in process'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/util/threaded_job_queue.rb:50:in `loop'
C:/opscode/chef/embedded/lib/ruby/gems/2.0.0/gems/chef-12.5.1-universal-mingw32/lib/chef/util/threaded_job_queue.rb:50:in `block (2 levels) in process'

Does this mean there is some EOFError in protocol.rb ? But that's part of the community cookbook, which I thought looked pretty solid.

2

2 Answers

0
votes

When chef-client starts up, one of the things it has to do is download the cookbook code it is going to use for the run. This is cached locally to speed future runs, but the first time or when files change etc etc. This exception is saying that it is failing while trying to download the ewe-powershell cookbook from the server. This possibly means it was corrupted during the upload process. A quick first check would be to knife cookbook delete that one and re-upload it. If that fails I would start trying to figure out what file is failing to download. If you turn up Chef's log level (-l debug) you should see which URL it is trying to get, which will tell you which file is failing. With that in hand, try curl'ing it directly from Bookshelf (the service with Chef Server that handles file upload and download).

0
votes

These are due to networking errors. They're not client bugs, and cannot be fixed by deleting and reuploading cookbooks.

There's some information here in a gist I made:

https://gist.github.com/lamont-granquist/e25af8f50cb4ae4f8050

Also more information:

https://discourse.chef.io/t/eoferror-when-decrypting-data-bags/5716/2 https://github.com/chef/chef/issues/1881

EOFErrors have been tracked down to broken proxies, PMTU issues (VPNs or filtering ICMP dest unreachable), jumbo frames, buggy TCP offload in networking drivers/hardware, Xen bugs, etc.

You should be able to replicate the problem on the command line if you use large enough packets. If your chef server is chef.example.com then you can send a large ping packet on windows with:

ping chef.example.com -f -l 9999

You can also send a large ping package or large GET request on unixes:

ping -s 9999 chef.example.com
curl http://chef.example.com/`printf '%*s' 9999 '' | tr ' ' 'a'`

Approaches like that can help replicate the issue completely outside of any chef-related software and show that its the network. The ultimate fix will have to come from spending the time to debug the underlying network problem. In this case, it sounds like the network magically fixed itself.

(and keep in mind by "the network" I include the host TCP stacks and network drivers on either end including any virtualization technology that network communication has to pass through--"the network" in the broadest sense possible...)