What are the general rules to when you should split resources into a separate subnet within a VPC?
Most of the articles I've found on the internet revolve around public/private subnets, but they weren't too in-depth. For example, there are many things you may consider private: ec2(s) behind ELB, databases, ... and things you may consider public: ELB(s), NAT instance(s), Bastion(s), ...
Should they all go in 1 big public subnet and 1 big private subnet? If not, what is the recommended approach?
First off, you need one subnet for each Availability Zone. I would not deploy a production environment without a minimum of two AZs (preferably three).
Secondly, I would use subnets to divide your application into coarse “tiers”. Public and private subnets at a minimum (one per AZ). Public-facing load balancers in the public subnets, servers in the private subnets. If you want to get more fine-grained you divide into more tiers like a traditional network (Public, Web DMZ, Database).
One thing I would keep in mind is how growth will affect things. Auto scaling groups can grow quite large. If you use Lambda functions in a VPC, you can easily have thousands of concurrent Lambdas eating up the IPs in your subnet. Container networking with EKS consumes a lot of IPs. If you mix Lambdas with auto scaling groups in the same subnet, you can have a nasty collision.
