Conditional validation in yup still validates fields that should not be validated

Question

I have a complex object that contains sub-objects. I only want to validate the sub-object if a condition is set.

To strip this down to a super basic example, lets sa