You are getting the validation error because the rel attribute has a list of values that it is expecting (see http://www.w3schools.com/html5/att_a_rel.asp).
There is nothing you can do to fix this, Fancybox is using the attribute for a purpose for which is was not intended. There's nothing wrong with that. Web pages don't have to be 100% 'valid'.