I'm using the Chef provided AWS cookbook (v2.7.2) to create and attach an EBS volume on an Ubuntu Server 14.04.02 LTS HVM ec2 instance during the initial bootstrapped chef-client run. I'm hosting my own chef server (12.1.0-1_amd64). Using knife-ec2, I create the aws instance and bootstrap the runlist via the following command:
knife ec2 server create --region us-east-1 --availability-zone us-east-1a --node-name my-server --run-list "role[my-role]" --image ami-d05e75b8 --flavor c3.large --ebs-size 16 --ebs-volume-type gp2 --subnet subnet-mySubnetId --associate-public-ip --server-connect-attribute public_ip_address --ssh-user ubuntu --ssh-key my-pem --identity-file /path/to/my-pem.pem -y
However, when the recipe executes that creates and attaches the volume, the volume successfully creates, but fails to attach to my instance. The following error keeps occuring:
Aws::EC2::Errors::InvalidInstanceIDNotFound: The instance ID 'i-36a93b84' does not exist
The following code is in my recipe (taken from the example on the cookbook page):
include_recipe 'aws'
aws_ebs_volume "backup_volume" do
aws_access_key aws['aws_access_key_id']
aws_secret_access_key aws['aws_secret_access_key']
size 16
device "/dev/sdi"
availability_zone "us-east-1a"
action [ :create, :attach ]
end
I verified 'i-36a93b84' is the id of the instance knife-ec2 creates and the chef-client is executing on, so it does exist. Also, I validated that I'm creating the volume in the same availability zone as the instance.
I've read some posts about AWS's Eventual Consistency, but have no idea how to handle this during a bootstrapped chef-client run, since the recipe runs immediately after the instance is created.
Is there a way to create an additional mount at the time knife-ec2 creates the aws server, or am I trying to attach a volume in an unorthodox manner?
Chef client output during the recipe:
Recipe: my-cookbook::backupvolume
* aws_ebs_volume[backup_volume] action create[2015-07-17T11:57:05+00:00] INFO: Processing aws_ebs_volume[backup_volume] action create (my-cookbook::backupvolume line 49)
[2015-07-17T11:57:05+00:00] DEBUG: Providers for generic aws_ebs_volume resource enabled on node include: [LWRP provider aws_ebs_volume from cookbook aws]
[2015-07-17T11:57:05+00:00] DEBUG: Provider for action create on resource aws_ebs_volume[backup_volume] is LWRP provider aws_ebs_volume from cookbook aws
[2015-07-17T11:57:05+00:00] DEBUG: Instance ID is i-36a93b84
[2015-07-17T11:57:05+00:00] DEBUG: Instance's availability zone is us-east-1a
[2015-07-17T11:57:05+00:00] DEBUG: Created new volume vol-4842ecc4
[2015-07-17T11:57:05+00:00] DEBUG: Volume is creating
[2015-07-17T11:57:08+00:00] INFO: Volume vol-4842ecc4 is available
- create a volume with id= size=16 availability_zone=us-east-1a and update the node data with created volume's id
* aws_ebs_volume[backup_volume] action attach[2015-07-17T11:57:08+00:00] INFO: Processing aws_ebs_volume[backup_volume] action attach (my-cookbook::backupvolume line 49)
[2015-07-17T11:57:08+00:00] DEBUG: Providers for generic aws_ebs_volume resource enabled on node include: [LWRP provider aws_ebs_volume from cookbook aws]
[2015-07-17T11:57:08+00:00] DEBUG: Provider for action attach on resource aws_ebs_volume[backup_volume] is LWRP provider aws_ebs_volume from cookbook aws
[2015-07-17T11:57:08+00:00] DEBUG: Attaching vol-4842ecc4 as /dev/sdi
================================================================================
Error executing action `attach` on resource 'aws_ebs_volume[backup_volume]'
================================================================================
Aws::EC2::Errors::InvalidInstanceIDNotFound
-------------------------------------------
The instance ID 'i-36a93b84' does not exist
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:212:in `attach_volume'
/var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:70:in `block (2 levels) in class_from_file'
/var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:68:in `block in class_from_file'
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/my-cookbook/recipes/backupvolume.rb
49: aws_ebs_volume "backup_volume" do
50: aws_access_key aws['aws_access_key_id']
51: aws_secret_access_key aws['aws_secret_access_key']
52: size 16
53: device "/dev/sdi"
54: description "TEST BACKUP"
55: availability_zone "us-east-1a"
56: action [ :create, :attach ]
57: end
58:
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/my-cookbook/recipes/backupvolume.rb:49:in `from_file'
aws_ebs_volume("backup_volume") do
action [:create, :attach]
updated true
retries 0
retry_delay 2
default_guard_interpreter :default
declared_type :aws_ebs_volume
cookbook_name "my-cookbook"
recipe_name "backupvolume"
aws_access_key "********"
aws_secret_access_key "********"
size 16
device "/dev/sdi"
description "TEST BACKUP"
availability_zone "us-east-1a"
timeout 180
volume_type "standard"
piops 0
end
Here is the stacktrace:
Aws::EC2::Errors::InvalidInstanceIDNotFound: aws_ebs_volume[backup_volume] (my-cookbook::backupvolume line 49) had an error: Aws::EC2::Errors::InvalidInstanceIDNotFound: The instance ID 'i-36a93b84' does not exist /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.48/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.48/lib/seahorse/client/plugins/param_conversion.rb:22:in `call' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.48/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.48/lib/seahorse/client/plugins/response_target.rb:18:in `call' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.48/lib/seahorse/client/request.rb:70:in `send_request' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.0.48/lib/seahorse/client/base.rb:216:in `block (2 levels) in define_operation_methods' /var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:212:in `attach_volume' /var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:70:in `block (2 levels) in class_from_file' /opt/chef/embedded/apps/chef/lib/chef/mixin/why_run.rb:52:in `call' /opt/chef/embedded/apps/chef/lib/chef/mixin/why_run.rb:52:in `add_action' /opt/chef/embedded/apps/chef/lib/chef/provider.rb:175:in `converge_by' /var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:68:in `block in class_from_file'
