This is not a cut and dry question. A single node cluster gives you the lowest latency because all your data will reside in your single vm along with your application because you don't incur any additional costs in serialization and chattiness. Obviously, this defeats the purpose of Hazelcast which provides for you the features required for the CAP Theorem like High Availability. That aside, you simply may not fit all your data in a single node. From here, as you add more nodes, that latency may increase if your embedded app is accessing data from another node, especially if it has multiple round-trips. But, you will more likely increase throughput, thus the ability to process more concurrently, because you'll have more cpu and memory available across all your nodes.
Sizing your cluster will depend on the usual variables such as: available heap size, data (object) size, count, indexes, retention policy, and estimated growth. There are many other variables to be considered that will affect the size of your cluster.
Embedded mode provides 1/N of data localisation where N is the number of servers in the cluster and data locality gets smaller with increase in cluster size. Apart from that, there is no real performance benefit with Embedded mode.
With Client-Server, although all data is remote but servers operate in their dedicate environment i.e. not sharing JVM and other resources with application code. This also means you will not be forced to run an application instance in order to run Hazelcast server, which btw is the case with embedded mode.
On a side note: if low latency is the requirement then NearCache can be used in either of the deployment modes.
Determining cluster size depends on various factors: size of total data to be cached, backup count, required TPS and Latency, need Index for queries etc. The guide covers those aspects in detail.
