How can I troubleshoot Error Syncing Cookbooks:?

Question

When running chef-client on one of our nodes, we are getting an error at the end of Synchronizing Cookbooks: saying Error Syncing Cookbooks: with no further message. What can we do to further diagnose this error?

Answer 1

You can start by adding the -l debug flag to your chef-client run. This will write debug level logging to /var/log/chef/client.log

For example: [2019-02-22T19:16:16+00:00] DEBUG: HTTP 1.1 404 Not Found

Since this 404 error means that a file is not found on the server, we can login to the chef-server and see what file is not available. Run chef-server-ctl tail on the chef-server to tail the logs from all the running services. Run chef-client again on your client and you should get more messages on the chef-server.

Here is an example from the nginx access.log:

==> /var/log/opscode/nginx/access.log <== [22/Feb/2019:21:25:06 +0000] "GET /bookshelf/organization-c1573b41a76ea22b9eb7c36d939fcad4/checksum-bfacb2422a1078b91660f763f0842e8d HTTP/1.1" 404 "0.002" 173 "-" "Chef Client/12.19.36 (ruby-2.3.1-p112; ohai-8.23.0; x86_64-linux; +https://chef.io)" "127.0.0.1:4321" "404" "0.002" "12.19.36" "algorithm=sha1;version=1.1;" "dev" "2019-02-22T21:25:06Z" "" 1265

This lets us know that the specific checksum in the bookshelf is missing. There is no easy way to correlate this to the corresponding cookbook, but you can make a psql database query, to determine which cookbook is causing the problem. Get your psql credentials on the chef-server from the chef-server.rb file: cat /etc/opscode/chef-server.rb

psql query using the checksum from the 404 error message line:

select name from cookbooks, cookbook_versions, cookbook_version_checksums where cookbooks.id = cookbook_versions.cookbook_id and cookbook_versions.id = cookbook_version_checksums.cookbook_version_id and checksum = 'bfacb2422a1078b91660f763f0842e8d';

Now that we have the cookbook name, we were able to increment the cookbook version number and re-upload it to the chef-server. The client was then able to find the new cookbook version and the 404 error was resolved.