I'm ignoring the flow as mentioned in your question because I don't want to make it any more complicated. Concentrating on the fact that you are willing to split authentication and authorization, as commented here by one of the creators of IdentityServer:
IdentityServer is both an OAuth 2.0 and OpenID Connect implementation.
Yes – we recommend to use IdentityServer for end-user authentication,
federation and API access control.
PolicyServer is our recommendation for user authorization.
The article itself is worth reading, and their PolicyServer is 'the answer' to the problem.
The idea is quite simple, though there are different approaches possible. It's likely that the implementation will take some time because the free, open source version uses a local source, while you will probably want to use a centralized version. In which case you can consider to take a look at the commercial version first.
In short, how it can work. First the user is authenticated. Then the authorization middleware adds the claims to the user. Then the user is authorized. In code, something like:
app.UseAuthentication();
// add this middleware to make roles and permissions available as claims
// this is mainly useful for using the classic [Authorize(Roles="foo")] and IsInRole functionality
// this is not needed if you use the client library directly or the new policy-based authorization framework in ASP.NET Core
app.UsePolicyServerClaims();
app.UseAuthorization();
In this case, you can leave the authentication flow intact and 'opt-in' authorization for the user. To speed up performance you can use caching.
The sample works fine for the Mvc client, but when you extend it with a resource (api) it will not work out of the box. Because the access token will not contain the claims. You can implement delegation using extension grants to create a token containing the claims (based on scope / resource), but you can also consider to leave it to the resource itself to request the authorization for the user from the PolicyServer. As the resource knows best what claims are relevant.
In other words, do not redirect the user, use the PolicyServer endpoints to determine the authorization for the client / resource.