Retrieve List of AWS Config Rule Names using AWS Golang SDK

Question

AWS Config has a set of Managed Rules and I am trying to use the Golang AWS SDK to use the DescribeConfigRules API to retrieve the list of AWS Config Managed Rule Names and other details.

It seems like every request receives a response of 25 rules and a NextToken for the next set of results. What I am having trouble understanding is how do I use this NextToken to retrieve the next set of results?

Here is what I have so far.

package main

    import (
        "fmt"
        "log"

        "github.com/aws/aws-sdk-go/aws"
        "github.com/aws/aws-sdk-go/aws/credentials"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/aws/aws-sdk-go/service/configservice"
    )

    func main() {

        //Create an aws session

        sess, err := session.NewSession(&aws.Config{Region: aws.String("us-west-2"), Credentials: credentials.NewSharedCredentials("", "my-aws-profile")})

        // Create a ConfigService client from just a session.
        configsvc := configservice.New(sess)

        rules := (*configservice.DescribeConfigRulesInput)(nil)

        configrulesoutput, err := configsvc.DescribeConfigRules(rules)

        if err != nil {
            log.Fatal(err)
        }

        for _, rule := range configrulesoutput.ConfigRules {
            fmt.Println("Rule: ", *rule.ConfigRuleName)
        }

    }

The above code successfully prints the first 25 rules received in the response. However I am not sure how to use the NextToken received in the response to get the next set of results.

Sample Response.

ConfigRules: [
    {
      ConfigRuleArn: "ConfigRuleARN",
      ConfigRuleId: "config-rule-ppwclr",
      ConfigRuleName: "cloudtrail-enabled",
      ConfigRuleState: "ACTIVE",
      Description: "Checks whether AWS CloudTrail is enabled in your AWS account. Optionally, you can specify which S3 bucket, SNS topic, and Amazon CloudWatch Logs ARN to use.",
      InputParameters: "{}",
      MaximumExecutionFrequency: "TwentyFour_Hours",
      Source: {
        Owner: "AWS",
        SourceIdentifier: "CLOUD_TRAIL_ENABLED"
      }
    },
    { Rule 2 }, ....{ Rule 25}
  ],
  NextToken: "nexttoken"
}

Code extracts the rulenames from the response and output is as below.

Rule:  cloudtrail-enabled
Rule:  restricted-ssh
Rule:  securityhub-access-keys-rotated
Rule:  securityhub-autoscaling-group-elb-healthcheck-required
Rule:  securityhub-cloud-trail-cloud-watch-logs-enabled
Rule:  securityhub-cloud-trail-encryption-enabled
Rule:  securityhub-cloud-trail-log-file-validation-enabled
Rule:  securityhub-cloudtrail-enabled
Rule:  securityhub-cmk-backing-key-rotation-enabled
Rule:  securityhub-codebuild-project-envvar-awscred-check
Rule:  securityhub-codebuild-project-source-repo-url-check
Rule:  securityhub-ebs-snapshot-public-restorable-check
Rule:  securityhub-ec2-managedinstance-patch-compliance
Rule:  securityhub-ec2-security-group-attached-to-eni
Rule:  securityhub-eip-attached
Rule:  securityhub-elasticsearch-encrypted-at-rest
Rule:  securityhub-elasticsearch-in-vpc-only
Rule:  securityhub-iam-password-policy-ensure-expires
Rule:  securityhub-iam-password-policy-lowercase-letter-check
Rule:  securityhub-iam-password-policy-minimum-length-check
Rule:  securityhub-iam-password-policy-number-check
Rule:  securityhub-iam-password-policy-prevent-reuse-check
Rule:  securityhub-iam-password-policy-symbol-check
Rule:  securityhub-iam-password-policy-uppercase-letter-check
Rule:  securityhub-iam-policy-no-statements-with-admin-access

End Goal: Using golang AWS SDK, extract the AWS Config Managed Rule details and put it in an excel format using Excelize to review which AWS Config rules we want enabled.

Thanks for your help in advance.

---New based on @Adrian's comment and doc reference---

As per doc

type DescribeConfigRulesInput struct {

    // The names of the AWS Config rules for which you want details. If you do not
    // specify any names, AWS Config returns details for all your rules.
    ConfigRuleNames []*string `type:"list"`

    // The nextToken string returned on a previous page that you use to get the
    // next page of results in a paginated response.
    NextToken *string `type:"string"`
    // contains filtered or unexported fields }

So here is what I am trying. Specifying nil should give me back all rules. nextToken is blank string for the first call.

configsvc := configservice.New(sess)
rules := (*configservice.DescribeConfigRulesInput)(nil)
nextToken := ""
rules.SetNextToken(nextToken)
getConfigRulesFunc(configsvc, rules)

//getConfigRulesFunc function

func getConfigRulesFunc(cfgsvc *configservice.ConfigService, ruleset *configservice.DescribeConfigRulesInput) {

    configrulesoutput, err := cfgsvc.DescribeConfigRules(ruleset)

    if err != nil {
        log.Fatal(err)
    }

    for i, r := range configrulesoutput.ConfigRules {
        fmt.Println("Rule: ", i, ""+*r.ConfigRuleName)
    }

    if *configrulesoutput.NextToken != "" {
        ruleset := (*configservice.DescribeConfigRulesInput)(nil)
        ruleset.SetNextToken(*configrulesoutput.NextToken)
        getConfigRulesFunc(cfgsvc, ruleset)
    }

}

Above code compiles fine but here the runtime error I believe because of nil.

configsvc type: *configservice.ConfigService
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13c7ed2]

goroutine 1 [running]:
github.com/aws/aws-sdk-go/service/configservice.(*DescribeConfigRulesInput).SetNextToken(...)
    /Users/user/go/src/github.com/aws/aws-sdk-go/service/configservice/api.go:12230
main.main()
    /Users/user/golang/awsgotest/awsgotest.go:26 +0x232

Pass it to the field of the same name in the Input for the next request. godoc.org/github.com/aws/aws-sdk-go/service/… — Adrian
@Adrian sorry to bother you with this. New to golang so still trying things out. I have updated the question with what I tried based on the doc. Starts with ---New based on Adrian's comment and doc reference--- in the question. Thank you for helping me. — Aakash
You didn't indicate which line it's triggered on but I'm guessing when it hits *configrulesoutput.NextToken != "" on the last page, configrulesoutput.NextToken is nil when you try to dereference it. — Adrian
Thank you for taking the time @Adrian. So with the code update above I receive the first 25 rows and a nexttoken. After printing the first 25 it checks if the nexttoken is "" and goes in the if and inside the if I have a couple of prints to show the nexttoken. The execution fails once I try to ruleset := (*configservice.DescribeConfigRulesInput)(nil) ruleset.SetNextToken(*configrulesoutput.NextToken) and recursively call getConfigRulesFunc(cfgsvc, ruleset). — Aakash
I would expect calling methods on a nil *configservice.DescribeConfigRulesInput to fail. You need to instantiate a value of it to use. — Adrian

Aakash Aakash · Accepted Answer · 2020-04-30T01:36:58

Ok, finally figured it out with the help of a very kind Alex Diehl via this ticket https://github.com/aws/aws-sdk-go/issues/3293 on the official aws-sdk-go repo.

I would still say the aws sdk for go definitely lacks simple examples for configservice at the least on recommended usage.

Here the code that works. This will also show how to use simple recursive function in go to use NextToken for pagination of api results that span multiple pages especially apis that do not have built in paginators.

Also note that DescribeConfigRules API does not list all AWS Managed Config Rules, only the Config rules enabled for your account.

package main

import (
    "fmt"
    "log"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/configservice"
)

var i int = 0

func main() {
    sess, err := session.NewSession(&aws.Config{Region: aws.String("us-west-2"), Credentials: credentials.NewSharedCredentials("", "my-profile")})
    if err != nil {
        log.Fatal(err)
    }

    //Create a ConfigService client from just a session.
    configsvc := configservice.New(sess)
    fmt.Printf("configsvc type: %T\n", configsvc)
    rules := &configservice.DescribeConfigRulesInput{}
    getConfigRulesFunc(configsvc, rules)
}

func getConfigRulesFunc(cfgsvc *configservice.ConfigService, ruleset *configservice.DescribeConfigRulesInput) {

    configrulesoutput, err := cfgsvc.DescribeConfigRules(ruleset)

    if err != nil {
        log.Fatal(err)
    }

    for _, r := range configrulesoutput.ConfigRules {
        fmt.Println("Rule: ", i, ""+*r.ConfigRuleName)
        i = i + 1
    }

    if configrulesoutput.NextToken != nil {
        fmt.Println("In if nexttoken is not empty")
        fmt.Println("Print NextToken: ", *configrulesoutput.NextToken)
        ruleset := &configservice.DescribeConfigRulesInput{}
        ruleset.SetNextToken(*configrulesoutput.NextToken)
        getConfigRulesFunc(cfgsvc, ruleset)
    }

}

Code in Bold were the ones giving me grief on how to use the NextToken based on best practices atleast for the go sdk for aws.

Retrieve List of AWS Config Rule Names using AWS Golang SDK

2 Answers