I have a form inside a dialog which I close by clicking on commandbutton with ajax ,
like this
I think you should take a look at PrimeFaces' RequestContext. This would help you trigger client-side code on the server side.