If you use on behalf access, only users which have access to the KeyVault can gain access through your API. In this case, you would give the API only "delegated" access to KeyVault, so it only works with a user.
User plus application access: The application accesses Key Vault on behalf of a signed-in user. Examples of this type of access include Azure PowerShell and the Azure portal. User access is granted in two ways. Users can access Key Vault from any application, or they must use a specific application (referred to as compound identity).
Ideally, you would give access to an AD Group, and add your users to this group. Giving individual users access is discouraged.
This tutorial also touches on what you desire.
This is for On-Behalf-Of Authorization scenarios which means that authorization is granted to a specific user only via a specific application. Without other Access Policies, the user cannot access the Key Vault without the app, and the app cannot access the Key Vault without the user. Most organizations will not use this feature, but I know that some have.
(Image taken from Azure Key Vault incorrectly creating Compound Identity)