It's not clear if you're talking about how to create a good dialog window or how to create a coherent look and feel for a huge application.
A pretty good reference for how do design an effective and clear UI is User Interface Design for Programmers by ... wait for it ... Joel Spolsky.
