To fix this I used inputAccessoryView.autoresizingMask = UIViewAutoresizingFlexibleHeight;
But of course this caused my textview to collapse.
So adding a constraint to the toolbar and updating it when I have to, or adding the constraint to the textview itself and update it worked for me.