Failing To Find Chef Community Cookbooks Using Chef 12 On AWS Opsworks

Question

The Problem

When running custom cookbooks on AWS OpsWorks a setup_failed status appears against the instance and the following in the failure log:

[2016-03-26T22:53:48+00:00] INFO: Started chef-zero at chefzero://localhost:8889 with repository at /var/chef
One version per cookbook
data_bags at /var/chef/runs/62832572-cb67-421a-8309-d831140d7b98/data_bags
nodes at /var/chef/runs/62832572-cb67-421a-8309-d831140d7b98/nodes

[2016-03-26T22:53:48+00:00] INFO: Forking chef instance to converge...
[2016-03-26T22:53:48+00:00] INFO: *** Chef 12.7.2 ***
[2016-03-26T22:53:48+00:00] INFO: Chef-client pid: 17842
[2016-03-26T22:53:49+00:00] INFO: HTTP Request Returned 404 Not Found: Object not found: chefzero://localhost:8889/nodes/test1.localdomain
[2016-03-26T22:53:49+00:00] INFO: Setting the run_list to ["recipe[my_cookbook::default]"] from CLI options
[2016-03-26T22:53:49+00:00] INFO: Run List is [recipe[my_cookbook::default]]
[2016-03-26T22:53:49+00:00] INFO: Run List expands to [my_cookbook::default]
[2016-03-26T22:53:49+00:00] INFO: Starting Chef Run for test1.localdomain
[2016-03-26T22:53:49+00:00] INFO: Running start handlers
[2016-03-26T22:53:49+00:00] INFO: Start handlers complete.
[2016-03-26T22:53:49+00:00] INFO: HTTP Request Returned 404 Not Found: Object not found: 
[2016-03-26T22:53:49+00:00] INFO: HTTP Request Returned 412 Precondition Failed: No such cookbook: apt

================================================================================
Error Resolving Cookbooks for Run List:
================================================================================

Missing Cookbooks:
------------------
No such cookbook: apt

Expanded Run List:
------------------
* my_cookbook::default

---

The OpsWorks stack is configured

• for Chef 12
• to use custom Chef cookbooks
• to use a Git repository for the custom cookbooks

The Custom Cookbook Repo

For the sake of finding the cause of the problem, a cookbook was created with a single recipe that runs apt-get update (which runs as expected under kitchen converge). The files were created using chef generate cookbook my_cookbook:

.
└── my_cookbook
    ├── Berksfile
    ├── chefignore
    ├── metadata.rb
    ├── README.md
    ├── recipes
    │   └── default.rb
    ├── spec
    │   ├── spec_helper.rb
    │   └── unit
    │       └── recipes
    │           └── default_spec.rb
    └── test
        └── integration
            ├── default
            │   └── serverspec
            │       └── default_spec.rb
            └── helpers
                └── serverspec
                    └── spec_helper.rb

The only altered files are:

my_cookbook/Berksfile

source 'https://supermarket.chef.io'    
metadata    
cookbook 'apt'

my_cookbook/metadata.rb

...
depends 'apt', '~> 3.0.0'

my_cookbook/recipes/default.rb

include_recipe 'apt::default'

Also, a Berksfile was added to the root of the repo

source 'https://supermarket.chef.io'
cookbook 'apt'

---

The Current Workaround

The current workaround involves downloading the dependencies using berks vendor and then copying the downloaded cookbooks into the root of the repo ie:

.
├── Berksfile
├── apt
└── my_cookbook

Is there a better solution (using AWS OpsWorks with Chef 12) that does not require explicitly storing and distributing community cookbooks with custom cookbooks?

Answer 1

I'm pretty sure this is how you are supposed to do it now. They removed a bunch of the automated berks integration between the Chef 11 and Chef 12 stacks for reasons I've never understood.

Answer 2

I have the same issue. It looks like it is supposed to use berk pack with s3 storage, but (as to me) it adds a other level of complexity (and is useful only with CI tools, where cookbook archive created by berkshelf would be convenient 'artifact'):

In Chef 12 Linux, Berkshelf is no longer installed on stack instances. Instead, we recommend that you use Berkshelf on a local development machine to package your cookbook dependencies locally. Then upload your package, with the dependencies included, to Amazon Simple Storage Service. Finally, modify your Chef 12 Linux stack to use the uploaded package as a cookbook source. For more information, see Packaging Cookbook Dependencies Locally.

From here.