The easiest way I found to circumvent this obvious flaw was to make two buttons. Make one of them Visibility.gone. Then simply change Visibility from the other one to gone and activate the first one by Visibility.visible.
I don't really like that solution, but it's faster and saner than the alternatives I found so far.