The configuration helps VMM only to know what is expected in the attribute returned from the LDAP server. VMM performs a fast lookup of the membership by using an attribute provided by the LDAP containing the membership. The config element where you can define if it is nested, all or direct only helps VMM to have a hint if it needs to perform further actions.
All depends on the values returned from LDAP. In your case Active Directory. Usually it would be memberOf and from https://msdn.microsoft.com/en-us/library/ms677943%28v=vs.85%29.aspx
memberOf
The memberOf attribute is a multi-valued attribute that contains groups of which the user is a direct member, except for the primary
group, which is represented by the primaryGroupId. Group membership is
dependent on the domain controller (DC) from which this attribute is
retrieved:
At a DC for the domain that contains the user, memberOf for the user is complete with respect to membership for groups in that
domain; however, memberOf does not contain the user's membership in
domain local and global groups in other domains.
At a GC server, memberOf for the user is complete with respect to all universal group memberships.
If both conditions are true for the DC, both sets of data are contained in memberOf.
Be aware that this attribute lists the groups that contain the user in their member attribute—it does not contain the recursive list
of nested predecessors. For example, if user O is a member of group C
and group B and group B were nested in group A, the memberOf attribute
of user O would list group C and group B, but not group A.
This attribute is not stored—it is a computed back-link attribute.
memberOf includes only the direct members. In combination with the configuration you have done VMM will only check the values inside of memberOf and does not perform any additional LDAP call.
Now I have too many options. Do you want to have the nested Groups for J2EE roles or within your application as you are calling VMM direct using API or or. I do not want to speculate to I leave it with the statement
memberOf returns the flat membership aka direct membership and VMM needs to perform additional calls to get the full info. By setting nested VMM will not perform any additional call