I have a simple class X, and set of templatized classes Y
The friend declaration page on cppreference.com specifies:
Friend declarations cannot refer to partial specializations, but can refer to full specializations
So as chtz said you can have a non-partial specialization friend.
Edit:
See also another answer on stackoverflow: https://stackoverflow.com/a/11046918/5776353