How to paginate over an AWS CLI response?

Question

I'm trying to paginate over EC2 Reserved Instance offerings, but can't seem to paginate via the CLI (see below).

% aws ec2 describe-reserved-instances-offerings --max-results 20                                                                                 
{
    "NextToken": "someToken", 
    "ReservedInstancesOfferings": [
        {
             ...
        }
    ]
}
% aws ec2 describe-reserved-instances-offerings --max-results 20 --starting-token someToken
Parameter validation failed:
Unknown parameter in input: "PaginationConfig", must be one of: DryRun, ReservedInstancesOfferingIds, InstanceType, AvailabilityZone, ProductDescription, Filters, InstanceTenancy, OfferingType, NextToken, MaxResults, IncludeMarketplace, MinDuration, MaxDuration, MaxInstanceCount

The documentation found in [1] says to use start-token. How am I supposed to do this?

[1] http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-reserved-instances-offerings.html

Answer 1

Looks like some busted documentation.

If you run the following, this works:

aws ec2 describe-reserved-instances-offerings --max-results 20 --next-token someToken

Translating the error message, it said it expected NextToken which can be represented as next-token on the CLI.

Answer 2

If you continue to read the reference documentation that you provided, you will learn that:

--starting-token (string)

A token to specify where to start paginating. This is the NextToken from a previously truncated response.

Moreover:

--max-items (integer)

The total number of items to return. If the total number of items available is more than the value specified in max-items then a NextToken will be provided in the output that you can use to resume pagination.

Answer 3

With deference to a 2017 solution by marjamis which must have worked on a prior CLI version, please see a working approach for paginating from AWS in bash from a Mac laptop and aws-cli/2.1.2

# The scope of this example requires that credentials are already available or
# are passed in with the AWS CLI command.  
# The parsing example uses jq, available from https://stedolan.github.io/jq/

# The below command is the one being executed and should be adapted appropriately.
# Note that the max items may need adjusting depending on how many results are returned.
aws_command="aws emr list-instances --max-items 333 --cluster-id $active_cluster"
unset NEXT_TOKEN

function parse_output() {
  if [ ! -z "$cli_output" ]; then
    # The output parsing below also needs to be adapted as needed.
    echo $cli_output | jq -r '.Instances[] | "\(.Ec2InstanceId)"' >> listOfinstances.txt
    NEXT_TOKEN=$(echo $cli_output | jq -r ".NextToken")
  fi
}

# The command is run and output parsed in the below statements.
cli_output=$($aws_command)
parse_output

# The below while loop runs until either the command errors due to throttling or
# comes back with a pagination token.  In the case of being throttled / throwing
# an error, it sleeps for three seconds and then tries again.
while [ "$NEXT_TOKEN" != "null" ]; do
  if [ "$NEXT_TOKEN" == "null" ] || [ -z "$NEXT_TOKEN" ] ; then
    echo "now running: $aws_command "
    sleep 3
    cli_output=$($aws_command)
    parse_output
  else
    echo "now paginating: $aws_command --starting-token $NEXT_TOKEN"
    sleep 3
    cli_output=$($aws_command --starting-token $NEXT_TOKEN)
    parse_output
  fi
done  #pagination loop