The first answer didn't totally solve my problem but gave me a huge hint.
Apple did add a private constraint to the accessory view, but you cannot find it in the constraint list of the accessory view. You have to search for it from its superview. It killed my a few hours.