aws ec2 request-spot-instances CLI issues

Question

Trying to start a couple of spot instances within a simple script, and the syntax supplied in the AWS documentation and aws ec2 request-spot-instances help output is listed in either JAVA or JSON syntax. How does one enter the parameters under the JSON syntax from inside a shell script?

aws --version

aws-cli/1.2.6 Python/2.6.5 Linux/2.6.21.7-2.fc8xen

aws ec2 request-spot-instances help -- at the start of "launch specification" it lists JSON syntax

--launch-specification (structure) Specifies additional launch instance information.

   JSON Syntax:

       {
         "ImageId": "string",
         "KeyName": "string",
         }, ....
         "EbsOptimized": true|false,
         "SecurityGroupIds": ["string", ...],
         "SecurityGroups": ["string", ...]
       }

I have tried every possible combination of the following, adding & moving brackets, quotes, changing options, etc, all to no avail. What would be the correct formatting of the variable $launch below to have this work? Other command variations -- "ec2-request-spot-instances" are not working in my environment, nor does it work if I try to substitute --spot-price with -p.

#!/bin/bash
launch="{"ImageId":"ami-a999999","InstanceType":"c1.medium"} "SecurityGroups":"launch-wizard-6""

echo $launch

aws ec2 request-spot-instances --spot-price 0.01 --instance-count 1 --type  c1.small --launch-specification $launch

This provides result: Unknown options: SecurityGroups:launch-wizard-6

Substituting the security group number has the same result.

aws ec2 describe-instances works perfectly, as does aws ec2 start-instance, so the environment and account information are properly setup, but I need to utilize spot pricing.

In fact, nothing is working as listed in this user documentation: http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-RequestSpotInstances.html

Thank you,

Answer 1

I know this is an old question, but in case somebody runs into it. I had the same issue recently with the CLI. It was very hard to get all the parameters to work correctly for request-spot-instances

#!/bin/bash
AWS_DEFAULT_OUTPUT="text"
UserData=$(base64 < userdata-current)
region="us-west-2"
price="0.03"
zone="us-west-2c"

aws ec2 request-spot-instances  --region $region --spot-price $price  --launch-specification "{ \"KeyName\": \"YourKey\", \"ImageId\": \"ami-3d50120d\" , \"UserData\": \"$UserData\", \"InstanceType\": \"r3.large\" , \"Placement\": {\"AvailabilityZone\": \"$zone\"}, \"IamInstanceProfile\": {\"Arn\": \"arn:aws:iam::YourAccount:YourProfile\"},   \"SecurityGroupIds\": [\"YourSecurityGroupId\"],\"SubnetId\": \"YourSubnectId\" }"

Basically what I had to do is put my user data in an external file, load it into the UserData variable and then pass that on the command line. Trying to get everything on the command line or using an external file for the ec2-request-spot-instances just kept failing. Note that other commands worked just fine, so this is specific to the ec2-request-spot-instances.

I detailed more about what i ended up doing here.

Answer 2

You have to use a list in this case:

"SecurityGroups": ["string", ...]

so

"SecurityGroups":"launch-wizard-6"

become

"SecurityGroups":["launch-wizard-6"]

Anyway, I'm dealing with the CLI right now and I found more useful to use a external JSON

Here is an example using Python:

myJson="file:///Users/xxx/Documents/Python/xxxxx/spotInstanceInformation.json"
x= subprocess.check_output(["/usr/local/bin/aws ec2 request-spot-instances --spot-price  0.2 --launch-specification "+myJson],shell=True)
print x

And the output is:

"SpotInstanceRequests": [
 {
"Status": {
            "UpdateTime": "2013-12-09T02:41:41.000Z", 
            "Code": "pending-evaluation", 
            "Message": "Your Spot request has been submitted for review, and is pending evaluation."
etc etc ....

Doc is here : http://docs.aws.amazon.com/cli/latest/reference/ec2/request-spot-instances.html

FYI - I'm appending file:/// because I'm using MAC. If you are launching your bash script using Linux, you could just use myJson="/path/to/file/"

Answer 3

The first problem, here, is quoting and formatting:

$ launch="{"ImageId":"ami-a999999","InstanceType":"c1.medium"} "SecurityGroups":"launch-wizard-6""

This isn't going to generate valid JSON, because the block you copied from the help file includes a spurious closing brace from a nested object that you didn't include, the closing brace is missing, and the unescaped double quotes are disappearing.

But we're not really getting to the point where the json is actually being validated, because with that space after the last brace, the cli is assuming that SecurityGroups and launch-wizard-6 are more command line options following the argument to --launch-specification:

$ echo $launch
{ImageId:ami-a999999,InstanceType:c1.medium} SecurityGroups:launch-wizard-6

That's probably not what you expected... so we'll fix the quoting so that it looks like one long argument, after the json is valid:

From the perspective of just generating valid json structures (not necessarily content), the data you are most likely trying to send would actually look like this, based on the docs:

{"ImageId":"ami-a999999","InstanceType":"c1.medium","SecurityGroups":["launch-wizard-6"]}

Check that as structurally valid JSON, here.

Fixing the bracing, commas, and bracketing, the CLI stops throwing that error, with this formatting:

$ launch='{"ImageId":"ami-a999999","InstanceType":"c1.medium","SecurityGroups":["launch-wizard-6"]}'
$ echo $launch
{"ImageId":"ami-a999999","InstanceType":"c1.medium","SecurityGroups":["launch-wizard-6"]}

That isn't to say the API might not subsequently reject the request due to something else incorrect or missing, but you were never actually getting to the point of sending anything to the API; this was failing local validation in the command line tools.