I need to create an modal dialog, which is to loaded from a nib file and should be displayed on a button click in the main window.
I can create a custom window in a nib
There are several ways to do this — and in fact two different kinds of modal dialog in OS X: application-modal and window-modal. Using the wrong one will annoy people. One is a sheet, which is attached to the window that it affects (save dialogs are an example of this). The other is application-modal, which blocks the whole application (open dialogs work this way, since they don't apply to any window that exists yet). Apple's sheet documentation should help get you oriented.