I’m coming in late to this conversation, but wanted to add my $0.02. As you mentioned GUI design is not MatLab’s strong suit. However, a lot can be done with it. If you have any specific questions I’d be happy to answer them. Also, realize that it is built on the java swing library so you can actually utilize those features. However, these features are not documented. There is a book title Undocumented Secrets of MatLab that provides a pretty decent discussion of the underlying swing components.
There are quirks to MatLab GUI, but if you can get over those then you’ll enjoy it. Also, as previously mentioned one of the more difficult hurdles is the concept of handles. Once you grasp that concept you will be 90% of the way there.