I found a workaround for my specific case, where I want to use the API only for me - I'm not making a product for anyone else. It's a bit of a hack.
Basically go to the Azure users page and open dev tools. Search for a user or something and you will see requests to https://graph.microsoft.com/beta/$batch.
Check one of those requests and it will have a request header like Authorization: Bearer <a_long_string>
.
You can use that same header to make graph requests yourself. I'm not exactly sure what permissions it has but it let me get /beta/users?$expand=manager
which I couldn't otherwise do. Note you can only get something like 100 users per request but the response includes a URL to get the next 100 so it's pretty easy to get them all.
The authorization token does not last long so it is only really useful for one-time export sort of tasks.