You'd have to subclass NSWindow
and do a custom window frame drawing. It's not only about a titlebar. It's about whole window frame (so you can, actually, put close/minimize/zoom buttons at the bottom if you wish).
A good starter is at "Cocoa with love" website.