I am trying to figure out how to define a function which takes the following two parameters:
You can't do what you are trying to do directly. It has nothing to do with reference types, it's because any constraints make T existential so it is impossible to satisfy them at the call site when you're referencing the protocol's metatype P.self: P.Protocol and an adopter C. There is a special case when T is unconstrained that allows it to work in the first place.
By far the more common case is to constrain T: P and require P: class because just about the only thing you can do with an arbitrary protocol's metatype is convert the name to a string. It happens to be useful in this narrow case but that's it; the signature might as well be register<T>(proto: Any.Type, obj: T) for all the good it will do.
In theory Swift could support constraining to metatypes, ala register<T: AnyObject, U: AnyProtocol where T.Type: U>(proto: U, obj: T) but I doubt it would be useful in many scenarios.