In the near future, I have to update a bigger XPage application (development time more than three years) now running on a Domino Server 8.5.3 FP6
to Domin
a big advantage with Bootstrap in comparison with oneUI is that there are many sites where snippets for controls / layouts / panels are being shared.
if you stick close to the oneUI defintion you are limited in your presentation options.
also Bootstrap is targetted against multiple clients (mobil / desktop / large) which oneUI is not. so if that is a requirement I definitely recommend bootstrap above oneUI.
there is a learning curve with Bootstrap but manageable.
with Bootstrap I hardly use dojo anymore and disable it in xpages whenever possible.
there are also limitations with bootstrap, e.g. motion awareness but you can overcome them with plugins. you would probably experience the same limitations with oneUI.
once you go bootstrap, you can't get back (to oneUI).
I have little experience with other front-end frameworks so I can not advice on that.