Why is AWS CloudFormation throwing "Encountered unsupported property InstanceGroups"?

Question

When I deploy the below AWS CloudFormation script, I am getting the following error: "Encountered unsupported property InstanceGroups"

I have used InstanceGroups in the past without any issues. Here is an example of how others using it: https://noise.getoto.net/tag/amazon-emr/

I am using EMR 5.17.0, which I have used to set up before.

{
  "Description": "Spark ETL EMR CloudFormation",
  "Resources": {
    "EMRCluster": {
      "Type": "AWS::EMR::Cluster",
      "Properties": {
        "Applications": [
          {
            "Name": "Hadoop"
          },
          {
            "Name": "Spark"
          },
          {
            "Name": "Ganglia"
          },
          {
            "Name": "Zeppelin"
          }
        ],
        "AutoScalingRole": "EMR_AutoScaling_DefaultRole",
        "BootstrapActions": [
          {
            "Path": "s3://somepath/scripts/install_pip36_dependencies.sh",
            "Args": [
              "relay==0.0.1"
            ],
            "Name": "install_pip36_dependencies"
          }
        ],
        "Configurations": [
          {
            "Classification": "yarn-site",
            "Properties": {
              "yarn.scheduler.fair.preemption": "False",
              "yarn.resourcemanager.am.max-attempts": "1"
            },
            "Configurations": []
          },
          {
            "Classification": "core-site",
            "Properties": {
              "fs.s3.canned.acl": "BucketOwnerFullControl"
            },
            "Configurations": []
          }
        ],
        "EbsRootVolumeSize": 10,
        "InstanceGroups": [
          {
            "Name": "Master",
            "Market": "ON_DEMAND",
            "InstanceRole": "MASTER",
            "InstanceType": "m5.2xlarge",
            "InstanceCount": 1,
            "EbsConfiguration": {
              "EbsBlockDeviceConfigs": [
                {
                  "VolumeSpecification": {
                    "SizeInGB": 100,
                    "VolumeType": "64"
                  },
                  "VolumesPerInstance": 1
                }
              ],
              "EbsOptimized": "True"
            }
          },
          {
            "Name": "Core",
            "Market": "ON_DEMAND",
            "InstanceGroupType": "CORE",
            "InstanceType": "m5.2xlarge",
            "InstanceCount": 5,
            "EbsConfiguration": {
              "EbsBlockDeviceConfigs": [
                {
                  "VolumeSpecification": {
                    "SizeInGB": 100,
                    "VolumeType": "gp2"
                  },
                  "VolumesPerInstance": 1
                }
              ],
              "EbsOptimized": "True"
            }
          },
          {
            "Name": "Task - 3",
            "Market": "ON_DEMAND",
            "InstanceGroupType": "TASK",
            "InstanceType": "m5.2xlarge",
            "InstanceCount": 2,
            "EbsConfiguration": {
              "EbsBlockDeviceConfigs": [
                {
                  "VolumeSpecification": {
                    "SizeInGB": 32,
                    "VolumeType": "gp2"
                  },
                  "VolumesPerInstance": 1
                }
              ],
              "EbsOptimized": "True"
            }
          }
        ],
        "LogUri": "s3://somepath/emr-logs/",
        "Name": "EMR CF",
        "ReleaseLabel": "emr-5.17.0",
        "ServiceRole": "EMR_DefaultRole",
        "VisibleToAllUsers": "True"
      }
    }
  }
}

When the CF script is loaded, it should create an AWS EMR cluster

Actually, I think I found the issue. InstanceGroup should be within the Instances object. — user422930

filipe filipe · Accepted Answer · 2019-06-17T17:05:08

Aws recommends that you set MasterInstanceGroup and CoreInstanceGroup under Instances

I give you an example of the Instances property of an EMR Cluster with Hadoop, Hbase, Spark, Ganglia and Zookeeper:

      Instances:
        Ec2KeyName: !Ref KeyName
        Ec2SubnetId: !ImportValue MySubnetPrivateA
        EmrManagedMasterSecurityGroup: !ImportValue EmrMasterSgId
        AdditionalMasterSecurityGroups:
          - !ImportValue EmrMasterAdditionalSgId
        EmrManagedSlaveSecurityGroup: !ImportValue EmrSlaveSgId
        AdditionalSlaveSecurityGroups:
          - !ImportValue EmrSlaveAdditionalSgId
        ServiceAccessSecurityGroup: !ImportValue EmrServiceSgId
        MasterInstanceGroup:
          InstanceCount: 1
          InstanceType: !Ref MasterInstanceType
          Market: ON_DEMAND
          Name: Master
        CoreInstanceGroup:
          InstanceCount: !Ref NumberOfCoreInstances
          InstanceType: !Ref CoreInstanceType
          Market: ON_DEMAND
          Name: Core
        TerminationProtected: false
      VisibleToAllUsers: true
      JobFlowRole: !Ref EMRClusterinstanceProfile
      ReleaseLabel: !Ref ReleaseLabel
      LogUri: !Ref LogUri
      Name: !Ref EMRClusterName
      AutoScalingRole: EMR_AutoScaling_DefaultRole
      ServiceRole: !Ref EMRClusterServiceRole
      Tags:
        -
          Key: "cluster_name"
          Value: "master.emr.my.com"

You can see the complete AWS template here.

Why is AWS CloudFormation throwing "Encountered unsupported property InstanceGroups"?

1 Answers