How do I fix this issue?
You don't need to. The range of values that the validator accepts for the rel attribute is limited.
Obviously Fancybox makes use of that attribute in a way that the validator does not appreciate, so it issues a warning an error; but this does not have to concern you.
Am I not writing the code out correctly?
Yes, you are.