Role based authorization is avoided these days and scope based authorization is preferred with something like service1.read, service2.full_access scopes. You could either: move role based authorization into each service and away from identity server, convert to scope based authorization, move role based authorization job to respective service rather than relay on identity server.
You can user reference token flow and invalidate token when changes occurs in your role/rights, this will help explaining it
