If I open a dialog like so:
$(\'\').dialog({
autoOpen: true
Have you tried this?:
$(window.parent).dialog('close');
I have never used the jQuery UI dialog, so I'm not sure that will actually work. It seems to me that you would need to maintain a reference to the dialog you created so that you can use it to close the dialog.
Note, you could also look for elements in the parent's DOM by:
$('#someParentDOMElement' , window.parent);
Of course, all of this is assuming that the site you load within the iframe is on the same domain as the parent document. If not, then the document in your iframe will not have access to the parent DOM at all.