I have a VPC in AWS account and there are 5 subnets associated with that VPC. Subnets are of 2 types - Public and private. How to identify which subnet is public and which is private ? Each subnet has CIDR 10.249.?.? range.
Basically when I launch an EMR in that subnet with lists of ec2SubnetIds , it says ***The subnet configuration was invalid: Provided subnet list contains both public and private subnet. Only one type of subnet is allowed.
How to recify this error.
The question is how to identify public subnets vs. private subnets, and the answer lies in what it means in AWS for a subnet of a VPC to be 'public' vs. 'private'.
Public subnets have a default route to an Internet Gateway; private subnets do not.
So, to determine if a given subnet is public or private, you need to describe the route table that is associated with that subnet. That will tell you the routes and you can test for a 0.0.0.0/0 route with a gateway ID of igw-xxxxxxxxxxxxxxxxx (as opposed to local).
You can tell if a subnet is public in the AWS VPC Console by reviewing the subnet's route table, for example:
You can also do this as follows for a given subnet ID, using the awscli:
aws ec2 describe-route-tables \
--filter Name=association.subnet-id,Values=subnet-0a123fc414ad5b999 \
--query "RouteTables[].Routes[]"
The output will look like this:
[
{
"DestinationCidrBlock": "10.0.0.0/16",
"GatewayId": "local",
"Origin": "CreateRouteTable",
"State": "active"
},
{
"DestinationCidrBlock": "0.0.0.0/0",
"GatewayId": "igw-0fca21fadaa22a1b2",
"Origin": "CreateRoute",
"State": "active"
}
]
Here, you can see a destination route of 0.0.0.0/0 with a target that is an Internet Gateway (its GatewayId is igw-xxxxxxxxxxxxxxxxx). This confirms that you are looking at a public subnet.
The best solution is to specify only a single subnet, so that you don't incur cross-AZ data charges.
There's no definite way to identify public and private subnets without looking at their routing tables: a public subnet will route to an Internet Gateway, while a private subnet won't. If you're creating your clusters via some program then maybe that's a reasonable check, but I wouldn't go there.
The best alternative is to give the subnet a name when you create it: something like "Private B" for a private subnet in availability zone B (us-east-1b, us-west-1b, whatever). If you're launching your cluster via the console that name should be shown in the list of available subnets (I haven't manually launched an EMR cluster in years, so don't know for sure).
Alternatively, you can give your subnets arbitrary tags. This is probably most useful if you're accessing them programmatically.
I found this to be rather challenging. I hope others find this useful. This approach uses aws cli and jq. As of mid 2021 I'm using current versions of both. To caveat, this is specific to only IPv4, and VPCs with only 1 internet gateway.
From a bash shell start off by identifying the VPC in which you are considering the subnets. In my use case I wanted the VPC hosting an EKS cluster.
VPC_ID=`aws eks describe-cluster \
--name ${CLUSTER_NAME} \
--query "cluster.resourcesVpcConfig.vpcId" \
--output text`
Find the internet gateway for that VPC
IGW_ID=`aws ec2 describe-internet-gateways \
--filters Name=attachment.vpc-id,Values=${VPC_ID} \
--query "InternetGateways[].InternetGatewayId" \
| jq -r '.[0]'`
Identify the public subnets
PUBLIC_SUBNETS=`aws ec2 describe-route-tables \
--query 'RouteTables[*].Associations[].SubnetId' \
--filters "Name=vpc-id,Values=${VPC_ID}" \
"Name=route.gateway-id,Values=${IGW_ID}" \
| jq . -c`
Show the public subnets
echo $PUBLIC_SUBNETS
List the private subnets by excluding public subnets
aws ec2 describe-subnets \
--filter Name=vpc-id,Values=${VPC_ID} \
--query 'Subnets[].SubnetId' \
| jq -c '[ .[] | select( . as $i | '${PUBLIC_SUBNETS}' | index($i) | not) ]'
