Check out the datatable.js file in Primefaces here. There is a javascript function called toggleExpansion.
Maybe you can override this function and call the original one when no row is expanded and show a message when another row is already expanded (and not call the original one).
Just an idea...