Laravel relationship based on custom attribute not working both directions

Question

Models Product and Category.

Product model has this custom attribute: getCategoryIdAttribute(). because the way to get category id is a bit complex.