I\'ve got a couple of really old MSDos based paint programs. They work on palette indexed image buffers. They have a number of spectacular shape drawing tools, brushes and e
IDA Pro handles nearly any binary, and is the gold standard for this kind of work, though contacting the author may prove helpful as well.
If you can convince the author to open source some or all of the program, that's the best method.
Beyond that, disassembly and debugging are going to be your friends. In some cases you may be able to decompile a piece of software if you know what it was compiled with originally. But disassemble it, and run it in a debugger to find the overarching structure, and look for the obvious functions that do the actual brush work.
It may seem daunting at first, but after some time you'll find it's surprisingly understandable.
-Adam