Take a look at CRM connection string examples. You can prompt logged in user for authentication. It creates a new connection to Common Data Service using the current logged in user via OAuth.
I wanted to connect to CRM to work with solutions for a project, so I used to CRM SDK for that by referring to PowerApps samples repo. And my project authenticates user using Azure B2B, and I wanted the same user to connect to CRM. So I used On-Behalf-Of flow.
I first got access token after authenticating user for the AD app. Then I used this token to get another token using UserAssertion for the target org (Ex: https://org12345.crm.dynamics.com/). You'll need to add Dynamics CRM permission like this Registering AD App. Now once I have the token for my target org, I use it like this to connect to CRM:
Uri serviceUrlDestination = new Uri(orgUrl + @"/xrmservices/2011/organization.svc/web?SdkClientVersion=8.2");
OrganizationWebProxyClient sdkServiceDestination = new OrganizationWebProxyClient(serviceUrlDestination, false);
sdkServiceDestination.HeaderToken = AccessToken;
CrmServiceClient serviceClient = new CrmServiceClient(sdkServiceDestination);
I don't know how to retrieve contact list of the logged in user as I used this for mainly for solutions, but once you are connected to CRM from the logged in user's context, I think you should be able to do other operations as well.