I don't have experience with XBAP, but my impression is similar to yours in that it's only restricted in terms of security when compared with WPF.
As for Silverlight: I recently used it to develop a complex agricultural telemetry monitoring user interface--it's more of a data visualization app than a traditional business app, but it has plenty of CRUD and there wasn't a single feature that I wanted to implement but couldn't because of my choice of technology. And that was SL2.
I'm sure you'd end up with more bells & whistles if you used XBAP, and if you can guarantee that your users are running .NET 3.5 (or is XBAP part of 3.0 too?) I'd go for it. Otherwise, I'd have no reservations in recommending Silverlight.
(Apologies if you wanted more of a feature comparison matrix--but I think you can find those at Microsoft.)
