I think there's some confusion in regards to the issue here.
The problem is that your Access token doesn't have the right permissions and this needs to be requested at the time of getting the access token.
Please see the Microsoft Graph API for memberof here : https://docs.microsoft.com/en-us/graph/api/user-list-memberof?view=graph-rest-1.0&tabs=http
To make this call depending on the type of token you're trying to use (Application/Delegated Permission)
Permission type Permissions (from least to most privileged)
Delegated (work or school account) Directory.Read.All, Directory.ReadWrite.All, Directory.AccessAsUser.All
Delegated (personal Microsoft account) Not supported.
Application Directory.Read.All, Directory.ReadWrite.All
Please refer to the tutorials/docs on how to do this.
It looks like you're interested in using the Auth Code flow, please note that there are many different flows that get different access tokens. Refer to the docs for more information on this :
Also please note that there are two AAD Endpoints as well, the v1.0 and v2.0 endpoints.
https://docs.microsoft.com/en-us/azure/active-directory/develop/authentication-scenarios - Auth Basics
https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow - Client Credentials v2.0
https://docs.microsoft.com/en-us/azure/active-directory/develop/v1-oauth2-client-creds-grant-flow - Client Credentials V1.0
https://docs.microsoft.com/en-us/azure/active-directory/develop/azure-ad-endpoint-comparison - Comparison of V1 and v2
https://docs.microsoft.com/en-us/azure/active-directory/develop/v1-permissions-and-consent - Permissions and Consent v1.0
https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-permissions-and-consent - Permissions and Consent v2.0