How do I set the set system services with a target replica set count of 5? If I increase the node count to 5, would that be enough?
Silver - Run the System services with a target replica set count of five
Not really, node reliability level is determined by the service fabric resource, so you need to use some way to modify the nodeType reliability level on the SF resource. Using https://resources.azure.com is probably the easiest way.
Just scaling vmss to 5 instances wont change it reliability level
The system services replica count are defined based on the reliability tier you chose on cluster creation.
Increasing the number of nodes will increase the durability of your services data, not the reliability, it might indirectly increase the reliability because you have more nodes to distribute the load and to takeover new replicas in case of a node failure, but it won't increase number of replicas required to meet the reliability tier.
The right way to do it is settings the reliability during cluster creation or updating the existing one.
To update the reliability of an exisring cluster, you have to run the Update-AzureRmServiceFabricReliability command
From the docs:
The reliability tier is used to set the number of replicas of the system services that you want to run in this cluster on the primary node type. The more the number of replicas, the more reliable the system services are in your cluster.
The reliability tier can take the following values:
Platinum - Run the System services with a target replica set count of nine Gold - Run the System services with a target replica set count of seven Silver - Run the System services with a target replica set count of five Bronze - Run the System services with a target replica set count of three