This code compiles fine:
{-# LANGUAGE MultiParamTypeClasses, FunctionalDependencies, FlexibleInstances,
UndecidableInstances, FlexibleContexts, EmptyDataDe
Type families are one-way: you can expand Rec a to its computed type, but you cannot (uniquely) go from the expansion back to Rec a. This makes applications of type functions unsuitable for instance signatures, as they can never trigger the instance to apply.
You could try instead:
instance Rec a ~ reca => Sel a s (b->(c,reca))
This means something else: it says any function b -> (c, reca) is an instance, and then when it has irrevocably matched, the compiler checks that Rec a ~ reca. But this might be good enough to do want in your case.