I\'m going to create a mail plugin for the OS X Mail.app application for some additional features.
I have no idea where to start as there is no official documentatio
There is no official supported way to build such a tool - you need to start trying to hook in to Mail.app without any official support.
If you want to persist on this sort of thing, then you'll need to understand how Mail.app internals work, which is a bunch of using the debugger and class dump to inspect libraries in other apps:
https://github.com/nygard/class-dump
You'll probably also want a way to inject code into other applications, for example:
https://github.com/rentzsch/mach_inject
And every time Apple update Mail.app you'll potentially need to redo everything :)